You need to read this post about Table Limits

If you don’t know how to check WebTrends table sizes by now, you’ll still find out eventually … the ugly way.

Applies to:  Software

It’s about time we wrote this post.  Overdue in fact.

You, WebTrends software user, need to know that this issue exists and you need to monitor it.  Starting now.

What is it?  After processing your logs, WebTrends stores data in a database.  The database is made up of linked tables.  Those tables have size limits.   It’s a fact of life, and table size limits are a good thing – they allow the program to work faster.

Why should you care?   You’re probably not aware that **IF** you fill up one of these tables the related reports can be wrong.  Most times you won’t notice or care.  Sometimes you will notice and you will care a LOT, as in the ugly examples below.

Ugly example #1:  Your end users contact you wanting to know about traffic to their expensive new microsite.  You know you’ve been collecting the data correctly because you triple-checked the tagging before and after launch.  So you open the Pages report and WebTrends tells you those pages don’t exist.  Those  expensive pages got no traffic at all, apparently.  Knowing how the CEO’s been obsessed with the new microsite, you call in sick indefinitely.

Ugly example #2:  Your marketeers contact you wanting to know about the effects of their latest paid search campaigns … were the visits from those keywords producing leads and purchases?  You know you have the data because you spent a long time checking all the destination URLs that the marketeers were about to upload to the SEs and you even clicked on a few of the ads after the campaign went live, to check the marker parameters.  Yet WebTrends can’t find a trace of most of those keywords, with or without marker parameters.  The whole campaign failed completely, you report.  But the marketeers have Google AdWords reports that show many clicks.   You hear the sounds of shouting in the distance and see glimmers of torchlight …

What has happened?  The WebTrends Pages table (ugly example 1) or Search Phrases table (ugly example 2) filled up at some point and aren’t accepting new entries any more.  Your new pages and fresh keywords aren’t being recorded.  They’re still in your raw data, but you’ve missed the chance to get them into reports.

Let me try saying it another way.  If the Page report’s Analysis table is filled up with URLs, new URLs won’t be recorded any more.   Yes, they’ll be counted anonymously in page view counts.  But their identity, the URL, gets discarded.  You won’t see the new URLs in any list of URLs.  Same for search terms, referring domains, whatever.

What do you need to do to avoid all this?

  • Check for filled-up tables in your profiles.
  • Determine which ones need to be watched, based on the size the tables are at already.
  • Set up a schedule for checking them every week or month.
  • Watch for tables that are getting close to filling up.
  • Expand the table sizes for those that are close, or take other actions such as starting a new profile.

 How do you check a profile’s table size situation?

  1. Edit the profile
  2. Open the Table Sizes tab – you’ll now see a biggish table that looks like this in 8.0 (looks slightly different in 8.1 and up).

tablesize-report3

  Look at the Analysis columns, identifying reports where:

  • There’s a colored ball-icon under Analysis Status — this means you’ve filled up a table
    -or-
  • The “Count” is close to the “Limit” in the Analysis Limit and Analysis Count columns.  For example, the Limit may be 100,000 and the Count may be 91,000.  Since the Count hasn’t reached 100,000 you haven’t yet filled up a table, but you’re pretty close.

What about the “Report” columns?

 Those have to do with the displayed tables, the ones WebTrends shows you.  We’ll talk about that more in the next post.

How do you increase table sizes?

There’s an “edit” icon over at the right for some of the tables.  Not all of them.  We’ll talk about it more in another post.

 This seems cumbersome.  Why doesn’t the program take care of all this?

Goooood question, so glad you asked.

WebTrends can’t have unlimited tables.  It just can’t.  The limits are meant to keep processing and retrieval speedy, so they’re a good idea.   Part of your job, as the human expert, is to evaluate which tables are important enough that they need to be made bigger.

Having said that from WebTrends’ perspective, there are still a few things to rant about, oh definitely.

  •  WebTrends utterly fails to tell the user how important table limits are.  It’s not in the documentation (we can’t find it) and it’s not in the UI.   I’m not sure it’s in the training either.
  • This means that every user will find out the hard way, except maybe those who use consultants or who read The WebTrends Outsider.   Having to be burned first is an irresponsible way for WebTrends to manage this.
  • It’s a credibility killer in the eyes of the end user.
  • A search in the forum on “table limits” produces five pages of mentions going back to 2004 and none of those mentions are happy.   It’s not like WebTrends doesn’t know it’s a problem.
  • At the very least, WebTrends should have the Alerts function cover this. That’s what computers are for.  And the Alerts framework is already in place, hey!
  • Two or three years ago WebTrends removed the warnings that appeared in the headings of reports when table limits had been exceeded for that report.  WebTrends replaced that very useful warning with … nothing.  Rumor has it that  the warning was removed because the tech support people were tired of dealing with calls about it.

End of rant.

Cool custom dimension: IP addresses

First, here’s the obligatory gripe.  How is it that WebTrends doesn’t offer IP address as a dimension???  What’s up with that?

Second, here’s the obligatory Outsider solution.  Make your own IP address dimension, of course!  It’s better than the Visitors report because this version will combine into one row all the visitors that have an IP address in common.

 Instead of this:

Top Visitors snip

You can have this:

IP address snip

This custom dimension works great if:

  • you are sessionizing based on the SDC cookie(s).   Your Visitors report will show people as something like “63.15.208.255-7203986748993” if they have a cookie and “72.181.194.189_Mozilla/5.0….” if they don’t.  Note that these visitor identifiers are the IP address, followed by a hyphen or an underscore, followed by other stuff.  That consistency – the IP address coming first – is the key to an easy extraction.
  • you are sessioning based on IP/User Agent.   Same thing happens there, the Visitors show as  IP_stuff.

If you are sessionizing based on a non-SDC cookie, you’ll have to check your Visitors report to see whether the cookie values contain the IP address, and adapt the method shown here.

Steps to create this custom dimension:

1.  Create a custom dimension within the Custom Reports area.  Name it.

I call mine “IP address from cookie” because I’m usually working with reports that are sessionizing based on cookie.  If your sessionizing is based on IP/UA, then it would be correct to call it “IP address.”  Why does the distinction matter?  If you’re pulling the IP from the cookie, the IP address in the cookie will be the IP address of the user at the time the cookie was originally set,  i.e. their first hit of their first visit.  If they have a laptop and are visiting from a different address, WebTrends will still just have the IP that’s in the cookie.  Unless, of course, for this purpose you create a profile that sessionizes on IP/UA just for the purpose of using this cool custom dimension.

2.  In the next screen, base the dimension on Visitor then click on the Advanced button.   The screen will refresh with all the Advanced options.  What you want to do is extract the cookie value from the Visitor info.  So, …

3.  Click the Regular Expression choice, and in the regex box enter this:

^([^-_]*)

That’s hat-parenthesis-bracket-hat-hyphen-underscore-bracket-star-parenthesis.  It’s a cute succinct regex using parentheses to denote an extraction of everything up to the first hyphen or underscore character.

4.  Save.

5.  Create a custom report with this dimension and whatever measures you fancy, such as visits and page views.

I always turn on “use interval data” for both visits and pageviews measures because I want to look for spikes in trend graphs in the final report.

 

Thank you to “SunnyG”on the WT user forum for giving me the idea for this post.