You need to read this post about Table Limits
It’s about time we wrote this post. Overdue in fact.
You, WebTrends software user, need to know that this issue exists and you need to monitor it. Starting now.
This is Part 1 of 3 or 4 on Table Limits. This post talks about the internal or Analysis table limits. Subsequent posts will talk about the external table limits and how to change table limits. We’ll also share what we know about future releases of WebTrends and how they might change the Table Limits situation.
What is it? After processing your logs, WebTrends stores data in a database. The database is made up of linked tables. Those tables have size limits. It’s a fact of life, and table size limits are a good thing – they allow the program to work faster.
Why should you care? You’re probably not aware that **IF** you fill up one of these tables the related reports can be wrong. Most times you won’t notice or care. Sometimes you will notice and you will care a LOT, as in the ugly examples below.
Ugly example #1: Your end users contact you wanting to know about traffic to their expensive new microsite. You know you’ve been collecting the data correctly because you triple-checked the tagging before and after launch. So you open the Pages report and WebTrends tells you those pages don’t exist. Those expensive pages got no traffic at all, apparently. Knowing how the CEO’s been obsessed with the new microsite, you call in sick indefinitely.
Ugly example #2: Your marketeers contact you wanting to know about the effects of their latest paid search campaigns … were the visits from those keywords producing leads and purchases? You know you have the data because you spent a long time checking all the destination URLs that the marketeers were about to upload to the SEs and you even clicked on a few of the ads after the campaign went live, to check the marker parameters. Yet WebTrends can’t find a trace of most of those keywords, with or without marker parameters. The whole campaign failed completely, you report. But the marketeers have Google AdWords reports that show many clicks. You hear the sounds of shouting in the distance and see glimmers of torchlight …
What has happened? The WebTrends Pages table (ugly example 1) or Search Phrases table (ugly example 2) filled up at some point and aren’t accepting new entries any more. Your new pages and fresh keywords aren’t being recorded. They’re still in your raw data, but you’ve missed the chance to get them into reports.
Let me try saying it another way. If the Page report’s Analysis table is filled up with URLs, new URLs won’t be recorded any more. Yes, they’ll be counted anonymously in page view counts. But their identity, the URL, gets discarded. You won’t see the new URLs in any list of URLs. Same for search terms, referring domains, whatever.
What do you need to do to avoid all this?
- Check for filled-up tables in your profiles.
- Determine which ones need to be watched, based on the size the tables are at already.
- Set up a schedule for checking them every week or month.
- Watch for tables that are getting close to filling up.
- Expand the table sizes for those that are close, or take other actions such as starting a new profile.
How do you check a profile’s table size situation?
- Edit the profile
- Open the Table Sizes tab – you’ll now see a biggish table that looks like this in 8.0 (looks slightly different in 8.1 and up).
Look at the Analysis columns, identifying reports where:
-
There’s a colored ball-icon under Analysis Status — this means you’ve filled up a table
-or- -
The “Count” is close to the “Limit” in the Analysis Limit and Analysis Count columns. For example, the Limit may be 100,000 and the Count may be 91,000. Since the Count hasn’t reached 100,000 you haven’t yet filled up a table, but you’re pretty close.
What about the “Report” columns?
Those have to do with the displayed tables, the ones WebTrends shows you. We’ll talk about that more in the next post.
How do you increase table sizes?
There’s an “edit” icon over at the right for some of the tables. Not all of them. We’ll talk about it more in another post.
This seems cumbersome. Why doesn’t the program take care of all this?
Goooood question, so glad you asked.
WebTrends can’t have unlimited tables. It just can’t. The limits are meant to keep processing and retrieval speedy, so they’re a good idea. Part of your job, as the human expert, is to evaluate which tables are important enough that they need to be made bigger.
Having said that from WebTrends’ perspective, there are still a few things to rant about, oh definitely.
-
WebTrends utterly fails to tell the user how important table limits are. It’s not in the documentation (we can’t find it) and it’s not in the UI. I’m not sure it’s in the training either.
-
This means that every user will find out the hard way, except maybe those who use consultants or who read The WebTrends Outsider. Having to be burned first is an irresponsible way for WebTrends to manage this.
-
It’s a credibility killer in the eyes of the end user.
- A search in the forum on “table limits” produces five pages of mentions going back to 2004 and none of those mentions are happy. It’s not like WebTrends doesn’t know it’s a problem.
- At the very least, WebTrends should have the Alerts function cover this. That’s what computers are for. And the Alerts framework is already in place, hey!
- Two or three years ago WebTrends removed the warnings that appeared in the headings of reports when table limits had been exceeded for that report. WebTrends replaced that very useful warning with … nothing. Rumor has it that the warning was removed because the tech support people were tired of dealing with calls about it.
End of rant.
9 comments
Hello,
Very good post, but there are documentation about this issue here:
http://product.webtrends.com/WRC/8.7d/ResourceCenter/rc/Library/PDF/AUG/Optimizing_Reports_Using_Table_Limiting.pdf
Regards
Wow Guillero you really know your way around the documentation.
However, when I talked about documentation, I was referring to the guides that help people set up WebTrends. The admin users guide for example could have a checklist of things to think about and things to monitor, ongoing. The document you pointed out is something people might find after they’ve run into the problem. It’s a band-aid. The issue of table limits needs to be in the top layer of documentation, i.e. broadcast. In the admin manuals. In the UI. Where people will see it before being burned.
I’ve noticed (on our installation anyway) that WebTrends will clear out space in certain tables that have reached their limit. My understanding is that it will not do this for the Pages or Visitors tables due to the number of reports that depend on these two tables.
I share your frustration though in the lack of alerts for when a table caps out though. We got burned on this last year on a few of our profiles!
Thank you for posting this. It’s not nice to let analysts get burned… they have long memories. I am personally happy to see new posts on the outsider. Keep up the good work!
Thanks for this monumental post. Sooner or later all who work with Webtrends software will encounter this issue either on our own installations or another we may be called to review. I could not agree more about the need to monitor table sizes and the need to have it documented in a clear, obvious way so that the topic is approachable ad meaningful to WT admins.
The good news is that because we’re using logs, and because we’re using the software version, we can always reanalyze and get the info back into our reports. But this can take a lot of time and elbow grease.
Again, thanks for this post – we need to float this one into the top 10 list, call to make this topic a part of training, and expose it to new users at every opportunity.
Another good post would be how to break up profiles in a meaningful way to better manage table limits and custom report limits.
Good point, Mitchell, it’s now on the list … unless YOU want to flesh that out?
Your presentations at WT Engage were amazing! Also, I’ve heard the table limits question is quite a big one internally there. Did you learn anything about progress?
Great post. You mentioned that this is likely 1 of 3 or 4 posts on this topic. I’d like to suggest that you incorporate the performance impacts of increasing table sizes. Webtrends technical support has warned me about this in the past, but then is vague about the specific impacts.
Good point. There is about 50 times more info coming out of WebTrends about the dangers of high limits than there is about the dangers of low limits. We’ll definitely incorporate the high limit info they provide.
Sean – At the WebTrends Engage conference last week, we learned that table limits are a much bigger deal than we’d thought. Apparently WebTrends gets bad feedback on it from big important customers as well as us tiny (still important) customers. And, in software 8.7, WebTrends has addressed it. First, the code in 8.7 allows much bigger table limits without as much impact on performance. Second, they have added table limits as an Alert topic — you can set up warnings to yourself about nearing table fill-ups.
We’re still going to finish the series since so many of us aren’t planning on going to 8.7 any time soon.
Leave a Comment