Recency Bias in Web Analytics
Summary:
Recency Bias in Web Analytics consistently lead to bad decisions and in worst case unfortunate overreactions! Overcome this widespread web analyst error in thought by first and foremost being aware of the bias existing - and install a rule set for critical thinking.
Biases are flaws in the way we reason and it cause us to make errors - and it is a verified fact that we place a disproportionate value on recent events (the Recency Bias) than to those events at the beginning or the middle of an observation. You know this in simple scenarios such as feeling really good about email as an online campaign channel, two days after the quarterly newsletter was sent out, which of course is a dreadfully biased feeling. This is not just a comical thinking defect, but a serious problem as the days of winging online marketing campaigns with great return on investments are long gone. The only way to win in online marketing today, being superior in comparison to your competitors, is to outthink them.
All that said, in a headline hungry society, people in general exacerbates the downside prospective of the recency bias, but it is even harder to beat this bias when we are presented with the positive effect of recency around the Internet as in e.g. Google Organic Ranking, Technorati Ranking etc. Therefore, more than ever, we should stress the importance of applying a rule set for critical thinking when working to overcome the Recency Bias.
Rule set for critical thinking
– overcoming the Recency Bias.
- Be critical
- Be empirical
- Be rational
Be critical
Be very critical towards your own conclusions, and this might sound undemanding and effortless, but it actually requires extensive experience in identifying the extent of one’s own ignorance. You become a whole lot less biased if you are intellectually humble – and if this is not within your character, practice this when you are by yourself. :-)
Note:
A great exercise is recalling previous beliefs that you once held strongly (and we all have that), but now reject and at best almost find embarrassingly naive. If you are out of suggestions, think about how you did online campaign attribution five years ago or how you did web site testing 5 years ago.
Be empirical
Let the data (the experiment itself) guide you, not the hypothesis you set to begin with. It is extremely important that you experiment where you do not know and that you make sure to read the numbers for what they are – we are all eager to force a complex situation (reality) into our somewhat simplified models.
Note:
A great way of building an empirical attitude is to simply ask your visitors what they think about a given hypothesis – it will also, quite interestingly conclude whether it was a question worth asking to begin with.
Be rational
Rationality and reason should be our key methods used to analyze any data set. And as a comment to that - the recency bias can result from an excess of information, disabling rational thought, where the solution should be to eliminate data that is not needed to conclude or cannot be acted upon. Information that cannot be acted upon simply distracts and should be avoided at any point. This is of course a general rule of thumb in Web Analytics.
Note:
Applying context is a fair initiative in being rational (not acting in a biased manner). And a great way to reduce the Recency Bias applying context is through data visualization like Sparklines - which you saw introduced with e.g. Google Analytics in the latest version. I wonder if this was to reduce the Recency Bias though – perhaps Avinash can elaborate on that when we meet next. :-)
Example on Recency Bias in Web Analytics
I honestly believe we on a regular basis are biases in our decision making, I also believe that we are unfairly presented with results in a biased environment, making it even harder to overcome. And a typical example of the Recency Bias in action is what we see in the use of Dashboards. Where my objection to most executive dashboards is that they tend to show only current values of a few metrics, taken completely out of context, and with little or no history applied to them.
Take the following Gauge (which to begin with, is a poor dashboard visualization form, in anything but real-time environments):
25% Paid Search visit to sale Conversion Rate (bear in mind, that this is in fact a real Dashboard item from a client of ours). To me - this sounds pretty damn fantastic – and it is presented in green and all, so it must be “good”. So what am I, as an executive, supposed to think now? Other than Paid Search is it!
Beyond the obvious – that this is clearly a way of presenting data in a recency biased environment - you as the analyst greatly intensify the recency bias and lets management feed on it by communicating at this level. Both the analyst and the executive are destined to overreact in an environment like that.
Now go back and apply critical thinking; be Critical, be Empirical and be Rational!
Conclusion
Awareness of the Recency bias is not always enough to stop us from making bad decisions or unfortunate overreactions, but by applying a rule set for critical thinking we provide an opening for us to be cautious in the conclusion we draw.
Side note:
Another really exciting bias (actually somewhat depressing though) is the one brought up by Nassim Nicholas Talib in his books Fooled by Randomness and in particular The Black Swan, where he concludes that we tend to focus on what we know, as opposed to what we do not know.
Cheers
Dennis
April 5th, 2008 at 15:27
Hi Dennis,
I just had a similar thought (I’ve thought about the recency bias - without knowing the expression or tying it to analytics, too but this post made me think of another bias in a similar way):
Think about going to school or your childhood..do you remember all those good memories, but don’t really remember the bad things you went through as much? Maybe we should call it the positivity bias ;).
I’m wondering if such a bias exists when it comes to looking at web analytics data, too:
What if you’re thinking about campaigns of the past and compare two campaigns (at a similar point in time) and one of them made your company a profit of 100,000$ whereas the other one made your company a loss of 100,000$ ?
Will most people be biased towards remembering the profitable campaign (while crowding out the campaign that made a loss)?
I know many people who bet money on sports (it’s legal here in Germany). They use ‘gut instinct’ and have never tracked a singificant sample size.
They have no idea if they’ve been making a profit or a loss over the last say 5 years they’ve been betting on sports (easily a game or more per day). However when somebody asks them they mention how they earn money on the side by betting on sports (Chances are like the vast majority of people they’re winning 50% of all games and make 91 cents on each such game and are losing 1$ on the other 50% of the games).
I really believe the reason why many of these people think they’re actually making money (other than not knowing well enough how that industry works) is that they are biased towards the games they won
(I hope I can judge this well enough by being around such people :-)).
I also think that we might be biased towards extreme events that have happened:
What effects could this possibly have when it comes to looking at web marketing campaigns?
Say somebody has been running a ton of PPC campaigns and most of them made a little loss, however they had a few PPC campaigns that performed EXTREMELY WELL and made them a FAT PROFIT.
SEO has been making them constant profits, but never anything that had that WOW-effect (like winning that bet which had odds of 1:15!)
Maybe that would biase some people towards PPC?
I think there are even other bias such as when you know somebody you are biased towards their opinion: Say 50% of all experts are of the opinion X, the other 50% are of the opinion Y. You have a good friend who’s an expert at that given topic who’s of opinion Y - I bet most people would be biased towards opinion Y!
(I often catch myself having such a bias and then try to think rationally: just because I know this expert and know he’s extremely smart and knowledgeable doesn’t mean he’s definitely right (even if he is most of the time)..if there are a lot of other experts in the field saying something else)
P.S.: Keep up the good work, this is the first time I’m posting, but I’ve been reading your posts regularly!
April 5th, 2008 at 15:37
I just read my post and realized just how long it got..sorry!;)
Unfortunately I also realized it might not be easy to follow my strange thoughts. So to sum it up:
- if we’re biased towards extreme events and/or positive events, we might think PPC is great if we had one home-run and 10 losses that didn’t hurt us much with PPC
I can imagine this might be an issue with viral marketing/link bait ideas…you lose a lot of money on trying to come up with a great viral marketing campaign/great link bait and fail at it a ton of times, but one time it works out great and you’re like WOW!!! - however over all you’re still at a loss)
P.S.: If we’re biased towards positive events in our childhood/teens, that has to be the reason why they call it “the good old times” lol
April 6th, 2008 at 2:36
Hi Patrick,
Thank you very much for the thorough and insightful response Patrick. AND It seems like we are very much on the same page in regards to biases in general. Good to see! :-)
N.B.
From time to time - I personally explain a lot of the Viral Marketing efforts with a Sensationalism attitude – which is essentially what’s called media bias - AND that is definitely a whole post by itself.
Cheers.
Dennis
April 6th, 2008 at 6:03
Does this mean you think for many companies trying to come up with that great viral marketing campaign doesn’t pay off - as they usually lose money (don’t make as much money as they could with an other investment) despite 1 in x times hitting a homerun?
I’m just curious - I’m still a beginner to online marketing/web analytics and didn’t mean to say that viral marketing (or PPC) for that matter didn’t pay off, it was just a random thought
January 6th, 2010 at 21:01
[...] a different way, Dennis Mortensen addressed the topic in his excellent blog post “The Recency Bias in Web Analytics,” where he points out the tendency to give undue weight to more recent numbers. He included a [...]
January 8th, 2010 at 16:05
Great post! My favorite analytics example of recency bias is all the tickets I’ve seen for “a big drop in traffic” on x date. 9 times out of 10, all it would take is widening the reporting window to realize that they never opened a ticket for the huge lift in traffic they got 6 weeks ago (which was the real problem) ;)
January 8th, 2010 at 16:13
Hi Jen,
Well put! But being brutally honest, I see myself fall into a recency bias from time to time.
Have a great weekend
d. :-)