The order in which WebTrends executes filters
Newbie users, you might want to avert your eyes because this topic could make you think you’ll just never master this program. There are so many ins and outs to know about, aren’t there.
(Actually, newbies, you’ll probably never need to know 99% of this stuff so don’t let it get you down. Most users never really need this information; it’s mainly for complex layered reporting. Wait. I lied. The one thing you need to know is in boldface at the end.)
The cool thing about the following list is that it explains so darn much. There IS an underlying logic. It DOES make sense. It’s not random, and it’s not out to get you!
WebTrends follows this order in processing filters and a few other things.
- Happens first: URL Search & Replace changes the URLs. (Details about ordering within URL S&R: as WebTrends takes in each successive log file line in a log, it looks for a match in the chronologically-sorted list of URL S&R’s and executes the first (oldest) one that matches. Once it has executed one S&R for a log file line, it goes to the next log file line. It only applies one URL S&R operation to any given log file line.)
- Next, individual hits are filtered in or out. When both Include and Exclude filters apply to the same hit, the Exclude wins. Includes cannot undo Excludes.
- Then, log file lines are sessionized (combined into visits) and whole visits are filtered in or out using profile-level Visit Filters. If an Exclude and Include apply to the same visit, the Exclude wins. Includes cannot undo Excludes. Visit filters cannot undo the previous Hit filters.
- Now the Main Analysis happens, including content group assembly, entry page identification, visit length calculations, identification of entry pages and referrers, etc.
- After the main analysis, Custom Report analysis happens. Within each Custom Report, visit filters are first, then hit filters. Notice that this order is the opposite of what happens at the profile level!
- Include filters in custom reports cannot restore something that has been excluded at the profile level, and custom-report Include Hit filters cannot restore something that has already been excluded by the custom visit filters.
- After custom reports are analyzed, and before the report is displayed to you, URL Rebuilding settings, if any, are applied. URL Rebuilding is not exactly a filter, but it’s important to know that it happens toward the end of the whole process and cannot interfere with filters.
- The template is applied. It filters out some reports from the display.
General cheat sheet:
- When Excludes and Includes conflict, Exclude always wins. An Include can never restore something that was Excluded at any point.
- At the profile level, hit filters are applied before visit filters. The order is the opposite for custom report filtering.
- URL Search & Replace happens before everything.
- URL Rebuilding and Templates happen after everything else.
The one thing you should be warned about:
- Because of all the above, you may get unexpected results in custom reports that have a dimension based on Visit and a filter based on Hit. For example, you may have a custom report that uses a hit filter to exclude PageA and has Entry Page as its dimension. You’ll find that your supposedly filtered-out PageA still appears in that custom report on Entry Pages. This is because the Entry Page for a visit is determined during the main analysis, step 4 above, and it is not changed by a hit filter at the custom level.In the above situation, if you had instead used a visit filter (removing all visits that have PageA as entry page) then PageA would not have appeared in the Entry Page custom report.
5 comments
Hi Rocky,
I just discovered your blog. It’s very interesting.
I was a bit surprised by the order of filters you indicate, and got confirmation in Advanced Configuration Guide p.34, that says that exclude filters are applied first, which I would argue is more logical.
Here’s an excerpt:
“You can use multiple filters of the same type (include or exclude) or combine different types
in the same custom report or profile. For example, if you want to determine how much of
your domestic Web server activity comes from organizations other than the military, you can
create an include filter for all U.S. user addresses and an exclude filter for all military user
addresses. WebTrends processes the include filters first and then the exclude filters.“
Jacques, thanks for the great comment. Consider yourself an honorary Outsider!
You’re right about what the documentation says. However it does conflict with older documentation and with what tech support has said, which is what we were relying on. So, we decided to set up some tests and it became clear that … it didn’t actually matter. We actually couldn’t come up with any ways to test it. We’d get the same results in our reports regardless of the order in which they’re applied.
In the end, we revised the posting to remove any references to the include-exclude order.
We also did some additional experimentation and ended up revising the posting quite a lot based on our results. Some of our posting still conflicts with what’s in the documentation, but we’re really sure we have it right.
If you can think of any situations where the order of exclude-include does matter, please let us know!
Hi Rocky,
Well, honored I am!
Now, you made me think here. My only merit with my comment was that I was taking the documentation at face value. You’re right, the more I think about it, the more I think the order doesn’t matter after all. I have in the past come across a case where it did, but can’t recall it (yeah, I’m past my mid-forties…). I will have to think it more throughly to find an example when the filter order is important, namely include first (if we are to believe WebTrends doc.), but now, I guess you’re right, since you did test it (which I should do instead of blindly believe vendors!).
I am pretty new to WebTrends and for the life of me I cannot get WebTrends MarketingLab 8.5a to give me a breakdown of subdirectory activity.
Say for instance I have a directory called ads at the root of my website and within the ads directory there are multiple directories and so on and so forth. The structure would look something like this:
http://www.mysite.com/ads/news/21
http://www.mysite.com/ads/revenue/8
etc, etc
So within WebTrends I setup an include hit filter that looks like this:
/ads/revenue/8
/ads/news/21
with the “include subdirectories” feature turned off.
I re-analyze my weblogs with only these two filters being applied and when the analysis is complete I check the Directories report and all I see is activity for http://www.mysite.com/ads
Can WebTrends report on subdirectory activity? I don’t have SDC so that’s not an option. I just assumed that by adding those hit filters I would get information of those locations witin the Directories report but that’s not the case.
Any suggestions would be so very very appreciated.
WebTrends 8.5a Marketing Lab Edition
cheers,
devin
Devin,
Great question. I think we’ll turn this into its own post or a Candy Jar item at some point, because it’s a nice-to-know.
You have to make a change in the configuration file itself. Are you using On Demand or software? This fix is for software users who can get at the *.wlp files.
I’m assuming you know how to find and edit the *.wlp file for this profile. If not, go here: http://www.webtrendsoutsider.com/2008/the-wlp-file-knowing-enough-to-be-dangerous/
In the [profile] section, find (or add) this line:
TopDirectoriesMaxDepth=
Set it to the number of levels of directories you want to display. Out of the box, the default is 1, which is why you’re seeing just the top level. So set it to 5, 10, whatever will do the job.
You should be aware that the numbers you see in the resulting report will not include activity in subdirectories. The stats are the numbers of people at that level, neither above it or below it. Hope that makes sense.
Leave a Comment