Tips, tricks, and pokes, just WebTrends Analytics

Applies to:  On-Premises (Software) version of WebTrends

Every On-Premises user should know how to make a test log.  It’s essential if you want to

• be absolutely sure your customizations are right
• just want a bunch of data to analyze, for your own WebTrends practicing

It usually takes only a few minutes to make a test log and feed it into a test WebTrends report profile.  In this post I describe one way to do it.

There are a lot of steps below, but if you understand how WebTrends and log files work together in a general way, you shouldn’t have any trouble and you can probably skim most of it.

Steps 6 and 7 in the second section are the value-adds of this post, the details that most people don’t think about.

You’ll need:

• Access to existing logs (ones you already feed to WebTrends)
• Admin access to WebTrends (the ability to create a new profile, Data Source, custom report, etc)
• A plain text editor, preferably one that will handle large files, for example TextPad.  NotePad will work okay if your logs aren’t very big.
• Knowledge of what you want to test, i.e. what parameters, URLs, whatever.
  • If you already know what you want in the log, you can build a short hypothetical visit in a fresh log by making copies of just one log file line, then editing the lines so they work as one visit.
  • Or go to your site and click around, making your own test visit that you can isolate into a test log and analyze by itself.  Often, just a few lines will be all you need to test.
  • If you mainly want bulk data as a WebTrends sandbox, do a lot of clicking.  Change browsers and erase cookies as you go in order to make yourself look like more than one visitor.  Use this as a core and copy, paste, jumble up to make the file bulkier.  You’ll straighten it out in Step 5.
  • For bulk data, you can also install a crawler and tell it to crawl your site, then use the resulting log.   HTTP Track is one crawler; see Wikipedia for others.  (contribute a little $$ to Wikipedia while you’re at it!)
  • Also using a crawler, you can feed it a list of the important URLs, then use the resulting log.

Steps to create a test log:

1. Find out where the current logs are located.  A basic way to find out is to edit a profile, identify the profile’s Data Source, then edit that Data Source  to see the file location info.
2. With the text editor, create a new empty file.
3. With the text editor, open one of the existing log files.  Open the one that contains your test visit or your crawler’s visit, depending on your basic approach.  If you want to build a hypothetical visit, you can copy just one line from any log.
4. Copy one or more data lines and paste it/them into a new blank file.
   1. Short hypothetical visit – Open any log file.  Copy one existing line and paste it several times into the new log, then edit the lines so they contain whatever URLs (etc) you are trying to check.  With this method, all the lines will have the same cookie and therefore they will all look like the same visitor, same visit.  Your resulting WebTrends report will be small, but it will be focused on exactly what you wanted to test.
   2. Test visit - Locate the lines from your test visit, if you did one, and copy  and paste them into a new file.  (After analysis, you’ll check to see if WebTrends shows exactly what you did in your test visit, no more and no less.)
   3. Bulk data – Grab the log containing your many-click visit or your crawler activity.  Copy and paste all the data lines.  To get more bulk, paste over and over again.  You probably should change the cookie values here and there to show a bigger number of unique visitors.
5. Change the time/date stamps on all the data lines in your file so they are strictly consecutive, although the time spacing between them doesn’t matter.  Check to make sure not a single line is out of order!  (I usually do this editing manually.) (Oh, and it goes without saying that none of your dates/time should be in the future.)
6. Copy and paste the #Fields line from an existing log into the first position of the new log.

   #Fields: date time c-ip cs-username cs-host cs-method cs-uri-stem cs-uri-query sc-status sc-bytes cs-version cs(User-Agent) cs(Cookie) cs(Referer) dcs-id

7. Copy the very last line of your new log and paste it into the very last position (you’ll have a pair of identical lines at first) then change its date to be 1 day later.WebTrends will not analyze this line but it MUST be there, with a time stamp at least a half hour later than everything else in the log.  I use a whole day later to be easier to see.
8. Save this new file.
   • The name of the file and the extension don’t matter to WebTrends; you’ll be telling the Data Source the exact path and name.
   • Save it in same place as the other logs or create a new folder, it doesn’t matter.  The important thing is that WebTrends can reach the file as well as not mistake it for a regular log and process it into your regular reports!

Steps to analyze the log:

1. Go into the WebTrends admin console
2. Go to Administration >> Data Sources and find the data source that corresponds to the one you’ve been messing with.  Create and edit a copy of it.
3. Edit the file location information to point to the file you just created.
4. Save and close the new Data source.
5. Go to the list of profiles and clone one or create a new one.  Name it appropriately.
6. Edit the new profile’s Data Source to be the new Data Source you just created.
7. Turn off all hit and visit filters (such as spider filters) especially if you used a crawler.
8. Save the profile, analyze it, and, most importantly, see if the result is what you expected.

Tags

Posts that WordPress seems to think are related :)

The ODBC driver is probably my favorite WebTrends feature.  I still haven’t bothered to use REST because ODBC does everything I want.

With the ODBC driver I can create a 1 click report in Excel that I can distribute to anyone in my organization. They don’t have to log in, they don’t have to figure out which profile has the report they want, they just need to click on the link and open a nice familiar Excel file that tells them exactly what they need to know.

The easiest way to use the ODBC driver is to use a program called Datalinks. But, if you’re like me and you don’t have a few thousand extra dollars lying around you can do much of what you need to do directly from Excel for free. It just takes a bit of hacking around in Excel. Here is what you need to do (this works in Excel 2003, it is probably slightly different in other versions) :

1. Install the ODBC driver. You can get to it from the “install components” link in the left nav of the admin area. If you don’t see it you may not have rights to use it. Ask your administrator to set you up.
2. In Excel go to Data –> External Data –> New Database Query. It will open up a data connection box. Choose new connection and OK. Then find the WebTrends ODBC driver on the list and select it.
3. Next you’ll have to fill out your connection details. If you’re using On Demand they should be filled in, if you’re using your own software then you can fill in the default login URL and your username and password to connect. Click “refresh” and choose your profile and template from the dropdowns. Click OK to save.
4. Now Excel will present you with a list of the available reports. The names will approximate the names you are familiar with from the normal interface, but they’ll be a bit different. Choose the report you want and click next. (NOTE: you can only pick ONE report. Sadly, the ODBC connection doesn’t do Joins.)
5. Continue through the wizard. You can filter and sort on pretty much whatever you want.
6. When you finish the wizard Excel will crunch for a while (the ODBC connection is not super speedy) and then it will ask you where you want to place the data in your spreadsheet. Choose a cell and the data will automagically appear. TaDa!
7. If you click in the data area and then click on the exclamation point in the External Data toolbar (or click on Data –> Refresh Data) Excel will reach back out to WebTrends and pull down fresh data. Neato!

So now the bad news. This is cool and all, but WebTrends has practically snatched defeat from the jaws of victory. The fatal flaw is that the ODBC driver doesn’t recognize “parameters” as used by Excel. Excel Parameters allow you to use values in a cell on the spreadsheet in the query itself. This really comes in handy when you’re trying to change dates. Luckily, there is a workaround of sorts in the form of Relative Date Macros. Relative date macros allow you to query for data for “yesterday”, or “last week” or “last month”. Here is how to use them in Excel:

1. Perform the steps 1-5 above. On the last screen of the wizard choose “edit this query in Microsoft Query”. If you’ve already retrieved the data go to Data –> Import External Data –> Edit Query.
2. You’ll get a Microsoft Query window. The easiest thing to do is press the “SQL” button in the toolbar and type in a “WHERE” clause after the existing SQL query. Here is an example query with a where clause that uses a relative date macro:
   
SELECT Browsers_0.Browser, Browsers_0.Hits, Browsers_0.Visits, Browsers_0.TimePeriod, Browsers_0.StartDate, Browsers_0.EndDate FROM Browsers Browsers_0
WHERE (Browsers_0.TimePeriod='%start=day-5;duration=day+1%')

   This query gets the browsers report for a single day 5 days ago. To see all the relative date macros search for “relative date macro” in WT help, it’s in the first link.
3. After you modify the SQL and close the little SQL window, close the Microsoft Query window as well. This will bring the data back into Excel. Format and you’re done!

Tags

Posts that WordPress seems to think are related :)

WebTrends has a feature called Content Groups, which is simply the ability to glom together groups of pages and treat the groups as reportable entities.  You can define and name content groups in the admin interface – no need to change tags or page code!

If you can report on Pages, you can do the same kinds of reports with Content Groups.

The concept is so simple that a lot of people just file it under “uh huh, got it” without thinking about what it can be used for.    It’s actually one of the biggest little features available, versatile and powerful.

Content Groups represent the opposite of the report granularity that gets touted and tooted all over the place.  Content Groups do NOT allow you to drill down to the littlest sub-detail.  Content Groups are used for drilling UP to the big picture.

Here are some ways to use content groups:

• Turn each of your KPIs into a content group.
  • Then use Content Groups as a secondary dimension in custom reports.  For the first dimension, use visit segmenting dimensions like new and return visitors, paid search keyword groups, referring domains, or entry pages.  The resulting report will show, for each segment, the KPI activity for those visits.  As one of our astute readers, Boston Matt, has pointed out, it acts like Google Analytics’ Goals.
  • Or use Content Groups as a primary dimension in custom reports that have important kinds of filters.  One interesting filter is based on search terms, and includes only search terms that are brand terms.  Another would be its flip side —- all search terms that do not contain brand terms.   If you’ve been paying attention to your search terms, you probably can think of quite a few subgroups that you’d like to examine for KPI activity.
• Define each vertical silo on your web site as a content group (leaving out the home page and other non-silo pages), and look for:
  • intensity within each silo (pages per visit, or pages per visit as a proportion of available pages in the silo)
  • degree to which visits contain more than one silo within a single visit (sum of visits to individual content groups, divided by number of visits to the site) – do you want people to visit more than one silo?
  • general proportionality (each silo’s visits or page views as a percentage of the total) - is it what you’d expect or want?
  • back-and-forth between silos (use Content Group Paths from Entry) – are visits staying entirely in a silo or flipping back and forth between two of them in a way that suggests confusion or perceived overlap?
• Put each horizontal level of your site into a content group, from the most general down to the deepest most visitor-committed level, and treat the collection of content groups as a funnel:
  • how many visits get to each depth point in your site?  Does it vary by source segment, or by entry page?  Which entry page or segment produces the largest segment of deep visits?

Here are some tips and details:

• A content group can be just one page.  It doesn’t always have to be a set of pages.
• You can use tags or the admin interface to make content groups.  We favor the UI because you can make new ones or adjust existing ones whenever you want.
• If a content group is a collection of pages, a “visit” to that content group means “a visit that included any of the pages in the group.”  If the visit hit ten different pages, all in Content Group A, the Content Groups report will display that activity as 1 visit, 10 page views for Content Group A.
• It’s handy to define one universal content group that will give you, in the Content Groups report, the total visits and total page views for the whole site.  You can get the same thing from the Overview Dashboard, but with a Universale content group in your CGs report you don’t have to flip back and forth.   The definition of that content group can be “*”.
• If you have a reasonable number of mutually exclusive content groups the Content Group Paths From Entry report is probably going to be really interesting.  “Reasonable number” means no more than five or six.  “Mutually exclusive” means no page can be in more than one content group – otherwise you get mush for results.
• As a dimension, Content Groups is a HIT dimension, not a visit dimension.  You can’t have it as a primary dimension and something like new vs returning visitors as a secondary dimension.  See more on these restrictions here.
• If you want Unique Visitors stats for content groups, you have to collect the Content Group of Interest parameter in each hit in the content group, and when analyzing you must have Visitor History for Content Groups turned on.
• If you want a content group drilldown report (content groups and content subgroups) you have to define them with parameters – WT.cg_n and WT.cg_s respectively.  I have no idea why WT doesn’t have this in the user interface definition capability.
• If you want Unique Visitors as a measure for content group reports, WebTrends requires they be defined using the WT.cg_n parameter (and a special switch in the Visitor History toggle set).

Finally, here’s a challenge for the advanced users.  The WebTrends user interface allows you to set up many collections and combinations of URL stems, query parameters, and query parameter values as content groups.  But there is one particular (somewhat minor) combination that the UI’s logic cannot handle.  Be the first to describe the one we’re thinking of, and we’ll send you a WebTrends Outsider t-shirt!

Tags

Posts that WordPress seems to think are related :)

This question came up in the WebTrends user forum and I’m turning it into a quick post, because all WebTrends users should know how to get an organic traffic report.  Here at Outsider Central we’ve been using this report for eons – since WebTrends added custom reporting ability back at the dawn of the WebTrends Enlightenment Age.

The question was “how can I get a report that lists only organic (natural) search terms that brought traffic to my site?  WebTrends has out of the box report on Search Phrase and Search Terms, but they combine paid search with natural listings.” and “WebTrends has an out of the box report for natural search terms, but it requires me to have put WT.srch=1 into the destination URLs of my paid search ads, and I haven’t done that.”

The answer is, briefly, “go ahead and use the Search Phrase or Search Term reports, but eliminate paid search traffic from them.”

The other answer is, if you have implemented WT.srch=1 in your paid search, just use the out of the box “Search Phrases (Organic)” family reports, and skip the rest of this post.

This requires:

• You have to create a custom report for this, so you have to have custom reporting ability.
• Your paid search traffic has to be distinguished from other traffic in some way, using something in the destination URL.  Not the referrer.  The destination URL.  Hopefully you have already set this up; we wrote a post about PPC tracking parameters a while back.

Here’s how.

1. Figure out if you already have markers in  your paid search traffic that identifies those visits as paid search.  (If you don’t, run, do not walk, to the computer to set up destination URLs properly in your paid search program.)
2. Figure out which of those parameters you’re going to use.  It doesn’t matter what the parameter value is; you are going to base a filter on the existence of that parameter.  Just choose one that only appears in paid search visits, and appears in all paid search visits.
3. Create a new custom filter -
   • type “Visit” for visits.
   • Filter on Entry Page.
   • Set the definition of the entry page to be anything, for example just an asterisk if you’re not using regular expressions.
   • Add a URL parameter to the definition
   • Set the Parameter Name to be the parameter that you identified in step 2.
   • Set the Parameter Value to be equal to anything, for example just an asterisk if you’re going to stay with the type Text.
   • Click on Done.
4. Create a new custom report -
   • Give it a name and title and category
   • Set the Primary Dimension to one of the search phrase/keyword dimensions.  I suggest Search Phrases.
   • Leave Secondary Dimension empty
   • Set your measures – Visits is the important one here but you might want to do more, such as Visitors and Page Views.   You could also make a separate copy of the report just for first-time or returning visitors, and so on.
   • In all these reports, I suggest turning “Include Interval Data” on, which means you’ll see a time-trend graph.  It will slow down the processing but it’s worth seeing sudden bumps in individual keywords.
   • In the Filters tab, find the filter you just created (that identified paid search traffic) and set it to Exclude.
   • Review the summary screen then save it.
5. You can also, of course, apply this filter to other dimensions like Pages reports (what pages were visits by organic traffic).
6. Activate the new custom report in the profile of your choice.  If you have software, you can create a new profile just for this and re-analyze old data, while also adding the report to one of your existing profiles to get data going forward.  If you’re a profile aficionado, you’ll understand this.  If not, this might be a little confusing, sorry.

Tags

Posts that WordPress seems to think are related :)

There’s one Content Group definition that we put in every single profile and that’s what this post is about – what it is and more importantly why it’s wonderful.  This particular Content Group just shows how many visits there were, overall.   We call it “Overall Visits” and its definition is simply an asterisk.  If somebody’s visit contained one page or a hundred, it will count as 1 visit here.

We load up our Content Groups reports with important site events all the time, which you probably do as well.  It’s so easy to make a Content Group that shows, for example, how many visits saw any variation of a form submittal acknowledgement page, or any kind of contact with customer service, or any mailto: event.  An of course there’s nothing preventing you from making a Content Group that consists of just one page … such as the purchase thank-you page or even just the home page. As we’ve said in other posts, Content Groups rock.

You might ask, why bother when you have the Overview Dashboard and other places where Visits are counted up?

Well, think about the value of having it right there in the Content Groups report.  By slapping the Content Groups report into Excel, you just have to add one formula that divides each Content Group’s visit count by the “Overall Visits” count, and you get “percent of all visits to the site that included this content group.”

In the screen shot below, all I did was copy directly from the Content Groups report screen and slapped it into Excel, then added a calculation column (in blue).    48% of visits saw the special offers content.  26% printed a coupon.  Only 21% got to a product detail page.  And so forth.

This is the one content group that deserves to be set up as Global when it’s first created.  Globality is a check box on the first configuration screen.

Tags

Posts that WordPress seems to think are related :)