Applies to:  Software

It’s about time we wrote this post.  Overdue in fact.

You, WebTrends software user, need to know that this issue exists and you need to monitor it.  Starting now.

This is Part 1 of 3 or 4 on Table Limits.  This post talks about the internal or Analysis table limits.  Subsequent posts will talk about the external table limits and how to change table limits. We’ll also share what we know about future releases of WebTrends and how they might change the Table Limits situation.  

What is it?  After processing your logs, WebTrends stores data in a database.  The database is made up of linked tables.  Those tables have size limits.   It’s a fact of life, and table size limits are a good thing – they allow the program to work faster.

Why should you care?   You’re probably not aware that **IF** you fill up one of these tables the related reports can be wrong.  Most times you won’t notice or care.  Sometimes you will notice and you will care a LOT, as in the ugly examples below.

Ugly example #1:  Your end users contact you wanting to know about traffic to their expensive new microsite.  You know you’ve been collecting the data correctly because you triple-checked the tagging before and after launch.  So you open the Pages report and WebTrends tells you those pages don’t exist.  Those  expensive pages got no traffic at all, apparently.  Knowing how the CEO’s been obsessed with the new microsite, you call in sick indefinitely.

Ugly example #2:  Your marketeers contact you wanting to know about the effects of their latest paid search campaigns … were the visits from those keywords producing leads and purchases?  You know you have the data because you spent a long time checking all the destination URLs that the marketeers were about to upload to the SEs and you even clicked on a few of the ads after the campaign went live, to check the marker parameters.  Yet WebTrends can’t find a trace of most of those keywords, with or without marker parameters.  The whole campaign failed completely, you report.  But the marketeers have Google AdWords reports that show many clicks.   You hear the sounds of shouting in the distance and see glimmers of torchlight …

What has happened?  The WebTrends Pages table (ugly example 1) or Search Phrases table (ugly example 2) filled up at some point and aren’t accepting new entries any more.  Your new pages and fresh keywords aren’t being recorded.  They’re still in your raw data, but you’ve missed the chance to get them into reports.

Let me try saying it another way.  If the Page report’s Analysis table is filled up with URLs, new URLs won’t be recorded any more.   Yes, they’ll be counted anonymously in page view counts.  But their identity, the URL, gets discarded.  You won’t see the new URLs in any list of URLs.  Same for search terms, referring domains, whatever.

What do you need to do to avoid all this?

• Check for filled-up tables in your profiles.  
• Determine which ones need to be watched, based on the size the tables are at already. 
• Set up a schedule for checking them every week or month. 
• Watch for tables that are getting close to filling up. 
• Expand the table sizes for those that are close, or take other actions such as starting a new profile.

 How do you check a profile’s table size situation?

1. Edit the profile
2. Open the Table Sizes tab – you’ll now see a biggish table that looks like this in 8.0 (looks slightly different in 8.1 and up).

  Look at the Analysis columns, identifying reports where:

• There’s a colored ball-icon under Analysis Status — this means you’ve filled up a table
  -or-
• The “Count” is close to the “Limit” in the Analysis Limit and Analysis Count columns.  For example, the Limit may be 100,000 and the Count may be 91,000.  Since the Count hasn’t reached 100,000 you haven’t yet filled up a table, but you’re pretty close.

What about the “Report” columns?

 Those have to do with the displayed tables, the ones WebTrends shows you.  We’ll talk about that more in the next post.

How do you increase table sizes?

There’s an “edit” icon over at the right for some of the tables.  Not all of them.  We’ll talk about it more in another post.

 This seems cumbersome.  Why doesn’t the program take care of all this?

Goooood question, so glad you asked.

WebTrends can’t have unlimited tables.  It just can’t.  The limits are meant to keep processing and retrieval speedy, so they’re a good idea.   Part of your job, as the human expert, is to evaluate which tables are important enough that they need to be made bigger.

Having said that from WebTrends’ perspective, there are still a few things to rant about, oh definitely.

•  WebTrends utterly fails to tell the user how important table limits are.  It’s not in the documentation (we can’t find it) and it’s not in the UI.   I’m not sure it’s in the training either.
• This means that every user will find out the hard way, except maybe those who use consultants or who read The WebTrends Outsider.   Having to be burned first is an irresponsible way for WebTrends to manage this.
• It’s a credibility killer in the eyes of the end user.
• A search in the forum on “table limits” produces five pages of mentions going back to 2004 and none of those mentions are happy.   It’s not like WebTrends doesn’t know it’s a problem.
• At the very least, WebTrends should have the Alerts function cover this. That’s what computers are for.  And the Alerts framework is already in place, hey!
• Two or three years ago WebTrends removed the warnings that appeared in the headings of reports when table limits had been exceeded for that report.  WebTrends replaced that very useful warning with … nothing.  Rumor has it that  the warning was removed because the tech support people were tired of dealing with calls about it.

End of rant.

Tags

Posts that WordPress seems to think are related :)

First, here’s the obligatory gripe.  How is it that WebTrends doesn’t offer IP address as a dimension???  What’s up with that?

Second, here’s the obligatory Outsider solution.  Make your own IP address dimension, of course!  It’s better than the Visitors report because this version will combine into one row all the visitors that have an IP address in common. 

 Instead of this: 

You can have this:

This custom dimension works great if:

• you are sessionizing based on the SDC cookie(s).   Your Visitors report will show people as something like “63.15.208.255-7203986748993″ if they have a cookie and “72.181.194.189_Mozilla/5.0….” if they don’t.  Note that these visitor identifiers are the IP address, followed by a hyphen or an underscore, followed by other stuff.  That consistency – the IP address coming first – is the key to an easy extraction.
• you are sessioning based on IP/User Agent.   Same thing happens there, the Visitors show as  IP_stuff.

If you are sessionizing based on a non-SDC cookie, you’ll have to check your Visitors report to see whether the cookie values contain the IP address, and adapt the method shown here.

Steps to create this custom dimension:

1.  Create a custom dimension within the Custom Reports area.  Name it. 

I call mine “IP address from cookie” because I’m usually working with reports that are sessionizing based on cookie.  If your sessionizing is based on IP/UA, then it would be correct to call it “IP address.”  Why does the distinction matter?  If you’re pulling the IP from the cookie, the IP address in the cookie will be the IP address of the user at the time the cookie was originally set,  i.e. their first hit of their first visit.  If they have a laptop and are visiting from a different address, WebTrends will still just have the IP that’s in the cookie.  Unless, of course, for this purpose you create a profile that sessionizes on IP/UA just for the purpose of using this cool custom dimension.

2.  In the next screen, base the dimension on Visitor then click on the Advanced button.   The screen will refresh with all the Advanced options.  What you want to do is extract the cookie value from the Visitor info.  So, …

3.  Click the Regular Expression choice, and in the regex box enter this:

^([^-_]*)

That’s hat-parenthesis-bracket-hat-hyphen-underscore-bracket-star-parenthesis.  It’s a cute succinct regex using parentheses to denote an extraction of everything up to the first hyphen or underscore character.

4.  Save.

5.  Create a custom report with this dimension and whatever measures you fancy, such as visits and page views. 

I always turn on “use interval data” for both visits and pageviews measures because I want to look for spikes in trend graphs in the final report.

Thank you to “SunnyG”on the WT user forum for giving me the idea for this post.

Tags

Posts that WordPress seems to think are related :)

Applies to:  Software

We haven’t been posting much because we have no time and no life.  The day job is 110% overwhelming us. 

But one thing that has saved a lot of time in the past few weeks has been being able to see all the configuration settings for our many custom reports.  In one big mad spreadsheet.  Since we have 1,297 custom reports at the moment, it helps.

Just to give you a glimpse of our daily life expressed as a spreadsheet, here’s our listing, in three columns and shrunk to about a jillionth of legible size.  No you can’t magnify it.  But read The Outsider for what we consider to be the best and brightest.

You can have one of these too, floor-to-ceiling if you wish.  Attached to this post is a quick and dirty (actually, somewhat slow and dirty) cscript that parses through all your WebTrends configuration files, or .ini files, and creates a tab-delimited text file with all the settings for all your custom reports.  It shows the name of each custom report, its GUID, and it looks up all the filter and dimension GUIDs and displays their names.  It was written by Mister Peabody who plans to polish it up some day, unless WebTrends Inc provides us with something slicker and better.

The tab-delimited file the script produces can be opened with Excel, producing a spreadsheet with these columns:

• Ini Section:  this column shows the section heading within the original config file, should you want to open the config file and find the report’s raw configurations quickly.  You’ll probably never use the information in this column.
• Report Name:  the name or title of the report.
• Report GUID:  the GUID or ID number assigned to the report by WebTrends and used internally by WebTrends.  It’s listed here because if you open a profile .wlp file, you’ll see only GUIDs for custom reports..
• Profile Category:  if you assigned the report to a WebTrends custom category, the category name appears in this column.
• Dimension 1:  the name of the report’s first or primary dimension, with the dimension’s GUID in case you ever want to look it up in the original ini file.  Also, the dimension information is annotated as to whether it’s a hit dimension or visit dimension, according to WebTrends, but only for custom dimensions you’ve created.  Out-of-the-box dimensions are identified as such (”OOTB”) but the config files provide no information about visits-vs-hit basis, so the output doesn’t show it.
• Dimension 2:  Same as above, but for the second or secondary dimension, if there is one.   
• Exclude Filters:  names of all Exclude filters applied to the report.  They will be marked ”hit” or “visit” depending on their basis.
• Include Filters: names of all Exclude filters applied to the report.  They will be marked ”hit” or “visit” depending on their basis.

Rightclick here and Save To Disk to get the “Dump of WebTrends Custom Report Configurations” script.

Here’s how to use this little script once you have it.   

1. Remove the “txt” filename extension
2. Edit the line that defines the location of the configuration files.  If your configuration files are in the default location and your program is in the default location, all you have to do is replace “X:” with the drive letter
3. Put the file somewhere accessible, like the root of the drive where WebTrends is installed
4. Browse to it with Windows Explorer and double click on it
5. Wait – it can take a while.

Details

• If a cell contains more than one configuration setting (for example, if the report has multiple filters), you’ll see all of them together in one cell, but each will be separated by a pipe character (vertical line |).  You can neaten up these cells by turning the pipes into within-cell line breaks, i.e. turn them into a list inside the cell.  Here’s how to do this for all of them at once:  do a global search and replace where “|” is replaced by Alt-0010 (that’s “hold down the Alt key and press – ON THE NUMERIC KEYPAD NOT ON THE ALPHA KEYPAD – zero zero one zero”).  Onscreen it will look like you’re going to replace the pipe with nothing, but Alt-0010 is an invisible “within-cell-line-break” character.
• Excel 2007 can handle cells with a lot of text, i.e. a lot of filter names.  Excel 2003 … not so much.
• If you look at the script code you’ll see that it’s fairly repetitive and adaptable.  That means we’ll use it for other similar functions in the future.  Watch this space.   But, remember, we’re just tossing it out there.  We don’t support or warranty it.
• Meanwhile, if you can improve it or replace it with something better, PLEASE DO. 
• If you have more than 1,297 different custom reports, let’s hear about it!

Tags

Posts that WordPress seems to think are related :)

Bing is [almost] here.

WebTrends is making it a lot easier to update your keywords.ini and browsers.ini fields.  Go to http://www.webtrends.com/support/browser-and-keywords-updater.aspx for the latest, which for keywords.ini is version 6.  (You’ll get a new keywords.ini and browsers.ini together.)

The referrer is “bing.com” and the query term is in the parameter “q.”  There’s another parameter, “form=” that you might want to consider reporting on also.  One value of  form= is QBLH, an interesting choice.  Look it up. 

If you’ve already customized your keywords.ini and you don’t want it to be overwritten, then you don’t want to use the updated file from WebTrends. So, here are instructions for modification of keywords.ini.

Step 1 – Find the file

There are usually two or three copies of keywords.ini in the typical WebTrends installation. Typical locations are:

/WebTrends/modules/analysis/engine/8.0d (8.1, 8.5, etc)
/WebTrends/storage/config/component/lookupdata/
/WebTrends/storage/config/engine/8.0d (8.1, 8.5, etc)

Open the first instance of keywords.ini with a proper text editor. By “proper” we mean something like TextPad rather than Notepad, because Notepad doesn’t play well with the system when the file is in use. With TextPad, you can [usually] take the risk of changing the file while WebTrends is running.

(Make a backup!)

Step 2 – Change the Engine list. Go to the end of the long Engine list. Find the last numbered entry. It’ll look something like this:

Engine367=Looksmart (if your last numbered entry is 367)

Add a line using the next number, like this:

Engine368=Bing

Step 2 – Add the specifications.  Go to the end of the specification list (groups of three or four lines).  The last one might look like this (if your last one is Looksmart)

[LookSmart]
ID1=looksmart.com
KeywordIndicator1=key=

Add a blank line and then this:

[Bing]
ID1=bing.com
KeywordIndicator1=q=


Step 3.  Save and close the file.

Step 4.  Make the same changes to other copies of keywords.ini in your installation.

 That’s all it takes. 

Postscript

We’ve done the manual change a little differently than WebTrends does.  We’ve chosen to use “bing.com” as the identifier, while WebTrends just uses “bing.”  It’ because we expect to see a bingaroo and a bingbang and who knows what else before long.

We’d like to note, being perfectionists as we are, that the Bing site has no information about itself that’s written for webmasters, analysts, etc.  Its “about” information is entirely a sell to potential searchers.

And, we noticed that The WebTrends Outsider is not properly indexed in Bing.  No big deal.  The same thing happened with Cuil, and look what happened to THAT.

Tags

Posts that WordPress seems to think are related :)

Applies to:  Software

You probably know that Google announced the following last month:  when somebody clicks on an organic search result item on a Google search results page, the referrer string for the landing page will now contain the rank of that original result item as it appeared on the Google results page, to that visitor.  So — if theyclick on a #6 position item, the referrer field of the landing page will contain a parameter “cd=6″ in addition to the normal “q=<search+term>” and other Google info.  Google’s plan is to have this organic rank URL parameter happening in all of its search results, eventually.  Right now, it’s fairly rare.  But here at The Outsider we’re on top of it anyway.

If you want to turn that “cd=<rank>” string into a custom dimension, this post tells you how.

With this dimension, you can make a custom report on keywords where each keyword has a list below it of ranks (1…2…3…), showing the number of visits that came from a listing at that rank.  The “list below it” is, in WebTrends-speak, called a secondary custom report dimension.  The list of keywords would be of course the primary custom report dimension for the report.

It’s pretty easy and actually uses the techniques we described in a recent post about using a subpart of a URL.  (In fact, it was our playing around with the new Google referrer field that led to that other post in the first place.)  In that post, we talked about using a subpart of a URL.  In this post, we’re going to use part of the referrer field.

There are two parts to this. 

First, you have to enable the display of referring page query parameters.  This is a simple hack to a config file (if you use the WebTrends software).  The hack is described in this post:  http://www.webtrendsoutsider.com/2010/the-referring-pages-report-getting-the-parameters-to-show-up/

Second, you have to create the dimension.  Set it up as follows:

1.  Create it and give it a name

2. On the next screen, base it on “Referring Page (initial in visit)”.  Using this basis will turn it into a visit-based dimension.  That’s how the WebTrends built-in search phrase dimension works too.  If you want, you can instead base it on “Referring Page (any in visit)” and it will be a hit-based dimension that will act in a different way.  See our post on visit versus hit dimensions for a little more information.

3. Click on the Advanced button to open the subpart dialog.

4.  Activate the Regular Expression button and type in this regular expression: 

cd=([0-9]*)

(what this does is … locates “cd=” then extracts any immediately-following numeral(s), and stops extracting when it hits something that is not a numeral.)

That’s it.  Now it’s up to you to make a new custom report with “Search Phrase” as the primary dimension and this as the secondary dimension.  We suggest NOT checking the box that says “Exclude activity without dimension data” for this new dimension, because it helps to see how incredibly sparse these new referrers are.  We also suggest that you filter out PPC traffic and that you filter-in the google.com referrers.

PostScript:

• As said above, Google hasn’t rolled this out completely.  You may not see results at first.  Some people estimate this is appearing in 10% of search results links on Google.  My experience is that it’s less than that.
• Google’s initial writeup of this hardly mentioned the rank parameter, but here’s the link.  http://analytics.blogspot.com/2009/04/upcoming-change-to-googlecom-search.html
• Google is doing this only for organic listings, not PPC.  If you use PPC ads, it would be a good idea to remove PPC traffic from this report.  We talked about identifying PPC traffic in another post with a bit more in yet another post on PPC reporting.
• We’d love it if you’d bother WebTrends with a feature request about adding the ability to base custom measures on the referrer.  At the very least, they will find out if it’s easy or hard for them to do, should they decide to do it.  We suspect they are already looking into it – give them encouragement.

Tags

Posts that WordPress seems to think are related :)