What is “Direct Traffic” ?

What is “direct traffic” in your referrer reports? Lots of things. More than you thought. Here are all the known reasons for the Direct Traffic classification.

Since Yahoo traffic is quickly turning into “direct traffic,” it seemed like a good idea to re-post this (see item 18).  This post is one of the most frequently-cited posts we’ve ever done.

“Direct Traffic” is a legacy label that no longer makes sense.   Once upon a time, in long-ago simpler days (approximately 2003), the absence of a referrer in log files could only mean that somebody typed your site’s name into the browser’s address window, or used a bookmark, which amounts to the same thing.

No longer.

Here’s our current list of reasons for an empty referrer field, a.k.a Direct Traffic, or as it should be called, “Unknown Referrer Traffic.” (Webtrends 10 does call it “Unknown Referrer,” I’m happy to say.)

Although this list will help you find ways to reduce your Direct Traffic to more realistic numbers (i.e. closer to just reporting on #1), Direct Traffic is still a mess.  Or, as Jacques Warren once said, “I, for one, never use Direct Traffic in my reports and analyses anymore. It’s full of unreliable crap.”

  1. Somebody really did type in the address, or used a bookmark to get to your page.
  2. They clicked on a link in an email.  (Recently, this includes webmail servers also.  They don’t pass “mail.yahoo.com” and the like any more.)
  3. The link was in a document, Excel workbook, or PDF
  4. The link was in Skype, GTalk, or AIM
  5. The link was in a mobile app and opened your site in a (mobile) browser.  For example, clicking on a URL in a tweet viewed in a Twitter app or client, or a mention seen in a Facebook app.
  6. The link originates at a secure (https:) page and your page is not secure (http) (this sometimes includes web mail servers)
  7. The link originates at a secure (https:) page and your page is secure (https:) for certain browsers, but not all
  8. Spiders and bots were working from a list of URLs from a previous crawl (this one mostly applies to server logs, rarely to SDC)
  9. Spiders and bots may be programmed to suppress the referrer information (this one mostly applies to server logs, rarely to SDC)
  10. The link to your site was in Javascript (this is mostly a problem with IE).  Javascript links to your site include those that open your site in a new browser window, or any kind of javascript redirect.  Many banners’ links are programmed this way.
  11. The link to your site is from within a Flash application (mostly a problem with IE) (there are a lot of ways to do this in Flash so there may be exceptions)
  12. Your landing page redirects to another page via a 302 temporaryserver-side redirect
  13. The link was on an intranet or some other web site behind a proxy or corporate gateway that was set up to strip referrers from requests
  14. The visitor has made changes to their browser that suppresses the referrer information
  15. The visitor has set one of your pages as their browser’s home page or a pinned tab.  This is especially a problem where you’re a big company and your employees have the site as their home page … but you should be filtering out your own company’s IP addresses in the first place.
  16. Another site has put your page content into an iFrame and coded the frame to suppress the referrer, in order to make it difficult for you to find out who is framing your content
  17. Certain A/B situations where visits directed to the B page group are redirected via 302  from the control page (A) to B too quickly for the tag to fire on A.  Check with your A/B vendor about whether this might happen with their product.
  18. Starting early 2014, they came from any Yahoo Search, if your landing page is http rather than https.  The complete anonymization of Yahoo Search traffic will be complete March 31 2014.
  19. Starting early 2014, they came from Bing secure search (secure search is optional to Bing users at this time, but may in the future follow Yahoo’s lead).

 

 

Any KPI Page Can Be Turned Into A Measure

Any tracked page or event can be a measure, i.e. in how many visits did this page happen? A small, super-valuable trick.

You know how “orders” and “revenue” show up in their own columns in WebTrends’ e-commerce reports?

You can get the same thing for any page or group of pages, not just orders.

Such as, a form.  All forms (as a group).    Each separate page in the checkout process.  The acknowledgement page that happens after a form is submitted.  A particular download.  A promotionally-oriented page.  The FAQ.  The site map page.  A particular folder or section of your site.  In other words, anything you think is important and that you want to quantify for different visit segments – you can display it as a measure.

Which means you can use these measures for report where the rows are:

  • a list of campaigns
  • a list of affiliate sites that were sources of traffic
  • visits coming from search terms that are brand words
  • first time and/or returning visitors
  • 1st visits, 2nd visits, 3rd visits, 10th visits, whatever

And, by using the calculated measures feature, you can display conversion rates from form view to form submission, or percentage of all visits that had a certain KPI page.  Since Webtrends allows 20 measure columns per custom report, so you can cram in a lot of great stuff.

The report segment below shows where we used this trick to show campaign results for six campaigns (six rows), showing campaign results in terms of:  visits that contained an application form, visits that contained the acknowledgement page after the application is submitted, and a couple of calculated conversion rates.

 

 

Here’s how to do this core, very central, very useful semi-trick.

Create a custom measure to show VISITS to the KPI page/group

Decide what page(s) (downloads, etc) you want to act as your KPI event(s).  As an example, it could be the page /apply/thankyou.aspx, which is the page seen after an application is submitted.

Open the WT admin screen and go to:

Report Configuration >> Custom Reports >> Measures >> New Measure

  • Give the measure a name that includes the word “visits” because this is going to be the visits version of the measure.  Same for the “column name.”  (Example:  “Visits with Submitted Applications”)
  • In “What to Measure,” “Value to Base On” is Query Parameter and “Parameter Name” is WT.ti.  (it could be ANY parameter with a text value; we chose WT.ti.  This is what we’re calling “hijacking.”)
  • In “When to Measure” choose “Hits that Match Specified URL.”  It’s the last choice in the dropdown menu that has “All Hits” as the top choice.
  • For “Do you want to sum this measure across the visit” choose “YES.”  This is the key part of getting this measure to count visits (i.e. more than once in a visit doesn’t register).
  • In the “URL Expression” window, fill in your KPI URL or regex string, following the same process you’d use if you were defining a Content Group.  For our example, this is where you’d enter “/apply/thankyou.aspx”.  Check the Regular Expression box if appropropriate.
  • In the currency window, choose “No Currency” but set the decimal places to 0.

Create a custom measure to show VIEWS of the KPI page/group

This is the same process as above with two differences, indicated by boldface.  The name is of course different, and “Sum Across The Visit” becomes “No.”  You can create a new measure from scratch or just clone the one you made in the previous step and make the needed changes.

Decide what page(s) (downloads, etc) you want to act as your KPI event(s).  As an example, it could be the page /apply/thankyou.aspx.

Open the WT admin screen and go to

Report Configuration >> Custom Reports >> Measures >> New Measure

  • Give the measure a name that includes the word “views” because this is going to be the views version of the measure.  Same for the “column name.”  (Example:  “Application-Thankyou Views)
  • In “What to Measure,” “Value to Base On” is Query Parameter and “Parameter Name” is WT.ti.  (it could be ANY parameter with a text value; we chose WT.ti)
  • In “When to Measure” choose “Hits that Match Specified URL.”  It’s the last choice in the dropdown menu that has “All Hits” as the top choice.
  • For “Do you want to sum this measure across the visit” choose “NO.”  This is the key part of getting this measure to count instances (i.e. more than once in a visit does get counted).
  • In the “URL Expression” window, fill in your KPI URL or regex string, following the same process you’d use if you were defining a Content Group.  For our example, this is where you’d enter “/apply/thankyou.aspx”.  Check the Regular Expression box if appropropriate.
  • In the currency window, choose “No Currency”.

 Use the measure in a custom report, as follows

Follow the usual steps for adding a measure to a custom report.  You’ll see your new measures in the dropdown list of available measures.

In the Measures screen, after you’ve chosen the measure from the dropdown list, you’ll see a little box where you choose “Method” (usually says “SUM”).   Set it to COUNT.  This is important.

 

Attach it to a profile and analyze.

 

 

Postscript 1:  If you use this in the “visit” version, use it with a dimension that does not change during a visit.   Your stats, if in the “visit” form, get a little crazy if you use a dimension that changes constantly during a visit.  See our post on hit and visit dimensions for a little more information.

 

Making Your First Custom Webtrends Report

For newbies – the basics of a custom report

This is for WebTrends newbies who are ready to try a custom report.  We think, we hope, that WebTrends users who have hesitated to tackle this ultra-valuable feature will find it far easier than they thought.  Often, the hesitation is simply due to terminology issues!  We’ll go slow.

A “report” is simply a table just like you see everywhere in WebTrends’ results .  It’s just rows and columns.  The rows have labels and are a list of things, like a list of page URLs or referrers.  The columns have labels and contain numbers that quantify the things in the rows, like number of visits or number of page views … per “thing” on the “list”.

Don’t read further until you have the above nicely fixed in your mental concept map.  List of things …  numbers for each thing on the list … the list of things goes down the side … the numbers for each thing go across.

Ready?

Dimensions

  • The “list of things” is called a “Dimension” in WebTrends.  WebTrends has a lot of ready-made dimensions, plus you can easily make additional ones, called custom dimensions.  

Examples of out-of-the-box dimensions:  Page URLs and titles.  Content Groups.  Referring sites.  Campaign names found in the WT.mc_id parameter.  Visitor’s cookie value.  Day of the week. New visitors and Return visitors. On-site search terms that appear in the parameter called WT.oss.

Examples of custom dimensions you can create:  Product names as found in your site’s “productID=” parameter.  Campaign names found in a parameter that has a name other than WT.mc_id.  Product colors as found in your site’s “color=” parameter.  On-site search keywords as found in a parameter called “searchterm” or something other than WT.oss.

Measures

  • The columns containing numbers are called “Measures.”  Again, WebTrends has a lot of them already made.  In addition to the out-of-the-box ones, you can of course make additional ones..

Examples of out-of-the-box measures:  Number of visits.  Percent of total site visits.  Number of views.  Viewing time.  Number of orders.

Examples of custom measures you can create:  Number of instances of the parameter “color” having the value “purple.”  Number of instances that contained the parameter “promocode=yes”.

Big point:

Dimensions and Measures ARE in fact a basic custom report!   You can add details like filters, but making a custom report is basically a matter of combining a dimension with one or more measures.

So …

Making a custom report in WebTrends goes something like this, once you have opened the Custom Reports >> Reports >> New Custom Report screen:

  1. You choose a dimension.
  2. You choose at least one measure.
  3. You give the report a name and save it into the custom report pool.
  4. You attach it to a profile.
  5. You make sure the template will allow the report to be displayed.
  6. You analyze some data.
  7. You look at the data.
  8. If you don’t like the custom report you modify it or you can un-attach it from the profile and delete it from the pool of custom reports.

That’s the basic structure, but it’s of course not the whole story.  Here are the two other essential things:

Filters

  • Use filters to make a custom report that shows data only for a subgroup of your overall data.   For example, you may want the custom report to display data only for first-time visitors, or visits from Google, or visits that included a purchase.

Examples of out-of-the-box filters:  Day of the week is Sunday.  Entry page is URL “xxxx.”   Visitors are Returning.  Campaign ID (from WT.mc_id) is “zzzzz.”  Visits that did NOT arrive from a search engine.

Examples of custom filters you can make:  Product page views where the product has the color parameter “purple” or “blue.”  Visits that contained at least one product page view where the product has the color parameter “purple” or “blue”.  Pages classified as error pages.  Visits that arrived through search terms that contained your company’s name.  On-site search terms that returned no results, i.e. that had a value of zero for the parameter than shows number of search results returned.

Using more than one dimension at a time

  • If you want, you can nest one dimension inside another, in a so-called 2-dimension Custom Report.  For example, you can nest the “Page URLs Viewed” dimension inside the “New vs Return Visitor” dimension.   The result would be a list of all the Page URLs Viewed by New visitors, followed by another list of Page URLs Viewed, this time by Return visitors.  All in the same report.  The “outside” dimension (New vs Return in this example) is called the Primary dimension and the inner nested dimension is called the Secondary dimension and the whole thing is a Two-Dimension Report.  By the way, when you’re ready, The WebTrends Outsider has a post with more details about the ins and outs of 2D custom reports.
  • You can take the concept further and have a drill-down report, which is the nesting of three or more dimensions.  This is a little more complicated to do than 2D reports, but not that much more.

Finally, there are some smaller details that you don’t have to worry about until you’re fairly comfortable making custom reports:

  • If you want your report to show a trend graph (over time) for a particular measure you have to tell WebTrends to do so, by checking the “use interval data” box.  Otherwise WebTrends will conserve database space by not storing the day-by-day info necessary for a trend graph.
  • If you have a trend graph, the first measure will be the one graphed in the default view.  Keep this in mind as you are adding your measures.
  • Check the box “Exclude activity without dimension data” if you don’t want a “None” row in your data for hits/visits that don’t fit the dimension.  We recommend not checking this box while you test your report, because the “None” row can help with troubleshooting.
  • If you use both Include and Exclude filters, remember that Exclude filters trump Include ones.

Having covered the basic concepts and structure of a custom report and hoping you’ll just want to jump in and feel your way through the setup of one, we want to add this:

The hard part of custom reports is deciding what should be the dimension and filtering. Really.  It is not always easy to translate some vague “I wanna know …” question into specifics of dimensions and filters.  If this stumps you, don’t be discouraged.  You will get better at it as your mind wraps itself around this way of thinking.

To get examples of some custom reports that have been explicitly described here in the Outsider, go to the Cool Custom Reports category.  A few of them are a little high-level but you’ll see custom report logic in action.

 

List Your Custom Report Configuration Settings in Excel

A cscript that reads the custom report settings in your WebTrends On Premises server and produces a summary of their configurations.

Applies to:  Software (On Premises)

This post is about a script that will produce an Excel spreadsheet that lists all of your custom reports and their configurations.  You can use it with On Premises software.  For On Demand, Webtrends has nothing similar.

Attached to this post is a quick and dirty (actually, somewhat slow and dirty) cscript that parses through all your WebTrends configuration files, or .ini files, and creates a tab-delimited text file with all the settings for all your custom reports.  It shows the name of each custom report, its GUID, and it looks up all the filter and dimension GUIDs and displays their names.  It was written by Mister Peabody who plans to polish it up some day, unless WebTrends Inc provides us with something slicker and better.

The tab-delimited file the script produces can be opened with Excel, producing a spreadsheet with these columns:

  • Ini Section:  this column shows the section heading within the original config file, should you want to open the config file and find the report’s raw configurations quickly.  You’ll probably never use the information in this column.
  • Report Name:  the name or title of the report.
  • Report GUID:  the GUID or ID number assigned to the report by WebTrends and used internally by WebTrends.  It’s listed here because if you open a profile .wlp file, you’ll see only GUIDs for custom reports..
  • Profile Category:  if you assigned the report to a WebTrends custom category, the category name appears in this column.
  • Dimension 1:  the name of the report’s first or primary dimension, with the dimension’s GUID in case you ever want to look it up in the original ini file.  Also, the dimension information is annotated as to whether it’s a hit dimension or visit dimension, according to WebTrends, but only for custom dimensions you’ve created.  Out-of-the-box dimensions are identified as such (“OOTB”) but the config files provide no information about visits-vs-hit basis, so the output doesn’t show it.
  • Dimension 2:  Same as above, but for the second or secondary dimension, if there is one.
  • Exclude Filters:  names of all Exclude filters applied to the report.  They will be marked “hit” or “visit” depending on their basis.
  • Include Filters: names of all Exclude filters applied to the report.  They will be marked “hit” or “visit” depending on their basis.

Rightclick here and Save Link As, to obtain the “Dump of WebTrends Custom Report Configurations” script.

Here’s how to use this little script once you have it.

  1. Remove the “txt” filename extension
  2. Edit the line that defines the location of the configuration files.  If your configuration files are in the default location and your program is in the default location, all you have to do is replace “X:” with the drive letter
  3. Put the file somewhere accessible, like the root of the drive where WebTrends is installed
  4. Browse to it with Windows Explorer and double click on it
  5. Wait – it can take a while.

Details

  • If a cell contains more than one configuration setting (for example, if the report has multiple filters), you’ll see all of them together in one cell, but each will be separated by a pipe character (vertical line |).  You can neaten up these cells by turning the pipes into within-cell line breaks, i.e. turn them into a list inside the cell.  Because the pipe character is tricky in Excel, here’s how to do it in Excel for all of them at once:  do a global search and replace where “|” is replaced by Alt-0010 (that’s “hold down the Alt key and press – ON THE NUMERIC KEYPAD NOT ON THE ALPHA KEYPAD – zero zero one zero”).  Onscreen it will look like you’re going to replace the pipe with nothing, but Alt-0010 is an invisible “within-cell-line-break” character.
  • If you look at the script code you’ll see that it’s fairly repetitive and adaptable (by you, if you care to try).  But, remember, we’re just tossing it out there.  We don’t support or warranty it.

Making a Report That Has Only One Row for “All Traffic”

Yes, you can have a report with only one row that’s just for ALL TRAFFIC. It’s useful.

It’s funny how often this need comes up in day to day analytics work.

You need a report that has only one row — and you want that row to contain all visits.  So simple.  Maybe you want to see a trend graph for one particular measure.  Your aim is to put this graph in a dashboard.

Here’s how:

  1. Go to Report Configuration > Custom Reports > Dimensions and click on “New”
  2. Name the dimension something like “All traffic (one dimension)” and make the Column Heading something like “All Traffic”
  3. Click on “Next” to get to the Based On tab.
  4. “Value to Base On” should be “Query Parameter”
  5. “Parameter Name” should be something that doesn’t exist.  I like the “nevergonnahappen” parameter name.
  6. Click on “Advanced” (see screen shot below)
  7. In the expanded Advanced screen, check the “Override Default String” box and, in the “New String” box, put the name that you want to appear as the label of the row.  Why are we doing this?  Because otherwise, that row will be called “None” and that will be confusing.  The most general name to put here would be “All traffic” or “trend.”  (It’s a pity that this row label override has to be set in the dimension definition and can’t be done when it is applied to a custom report.  To make good clear custom reports with concise row labels, you’ll have to create a separate one-row dimension for each measure you want to use it with … purchases, downloads, customer service form submits, whatever.  Oh well.  Try to compensate by being creative with the custom report name.)
  8. In the “When To Collect Data” tab, do “All Hits”  (unless you have a well-thought-out reason to do otherwise)
  9. Save

nevergonnahappen

 

Now apply this to a custom report.  Remember a couple of things when defining the custom report:

  • Never check the box “Exclude activity without dimension data” – that’s the None row and this dimension IS the None row!
  • If you are going for a trend graph to put into a dashboard, be sure to have Interval Data turned on for that measure.

Postscript:  Since this trick focuses on measures, did you know that practically any URL can be turned into a measure?  The measure would show a count of the times that URL was seen?  See our “Measures” posts using the Categories at the right.  There are several such posts; start with this one:  http://www.webtrendsoutsider.com/2008/how-to-display-any-kpi-page-as-a-measure-column/.  Don’t settle for the measures that Webtrends provides!