Recency Bias in Web Analytics
| posted by Dennis R. Mortensen Tuesday, April 1, 2008 |
 |
Summary:
Recency Bias in Web Analytics consistently lead to bad decisions and in worst case unfortunate overreactions! Overcome this widespread web analyst error in thought by first and foremost being aware of the bias existing - and install a rule set for critical thinking.
Biases are flaws in the way we reason and it cause us to make errors - and it is a verified fact that we place a disproportionate value on recent events (the Recency Bias) than to those events at the beginning or the middle of an observation. You know this in simple scenarios such as feeling really good about email as an online campaign channel, two days after the quarterly newsletter was sent out, which of course is a dreadfully biased feeling. This is not just a comical thinking defect, but a serious problem as the days of winging online marketing campaigns with great return on investments are long gone. The only way to win in online marketing today, being superior in comparison to your competitors, is to outthink them.
All that said, in a headline hungry society, people in general exacerbates the downside prospective of the recency bias, but it is even harder to beat this bias when we are presented with the positive effect of recency around the Internet as in e.g. Google Organic Ranking, Technorati Ranking etc. Therefore, more than ever, we should stress the importance of applying a rule set for critical thinking when working to overcome the Recency Bias.
Rule set for critical thinking
– overcoming the Recency Bias.
Be critical
Be very critical towards your own conclusions, and this might sound undemanding and effortless, but it actually requires extensive experience in identifying the extent of one’s own ignorance. You become a whole lot less biased if you are intellectually humble – and if this is not within your character, practice this when you are by yourself. :-)
Note:
A great exercise is recalling previous beliefs that you once held strongly (and we all have that), but now reject and at best almost find embarrassingly naive. If you are out of suggestions, think about how you did online campaign attribution five years ago or how you did web site testing 5 years ago.
Be empirical
Let the data (the experiment itself) guide you, not the hypothesis you set to begin with. It is extremely important that you experiment where you do not know and that you make sure to read the numbers for what they are – we are all eager to force a complex situation (reality) into our somewhat simplified models.
Note:
A great way of building an empirical attitude is to simply ask your visitors what they think about a given hypothesis – it will also, quite interestingly conclude whether it was a question worth asking to begin with.
Be rational
Rationality and reason should be our key methods used to analyze any data set. And as a comment to that - the recency bias can result from an excess of information, disabling rational thought, where the solution should be to eliminate data that is not needed to conclude or cannot be acted upon. Information that cannot be acted upon simply distracts and should be avoided at any point. This is of course a general rule of thumb in Web Analytics.
Note:
Applying context is a fair initiative in being rational (not acting in a biased manner). And a great way to reduce the Recency Bias applying context is through data visualization like Sparklines - which you saw introduced with e.g. Google Analytics in the latest version. I wonder if this was to reduce the Recency Bias though – perhaps Avinash can elaborate on that when we meet next. :-)
Example on Recency Bias in Web Analytics
I honestly believe we on a regular basis are biases in our decision making, I also believe that we are unfairly presented with results in a biased environment, making it even harder to overcome. And a typical example of the Recency Bias in action is what we see in the use of Dashboards. Where my objection to most executive dashboards is that they tend to show only current values of a few metrics, taken completely out of context, and with little or no history applied to them.
Take the following Gauge (which to begin with, is a poor dashboard visualization form, in anything but real-time environments):
25% Paid Search visit to sale Conversion Rate (bear in mind, that this is in fact a real Dashboard item from a client of ours). To me - this sounds pretty damn fantastic – and it is presented in green and all, so it must be “good”. So what am I, as an executive, supposed to think now? Other than Paid Search is it!
Beyond the obvious – that this is clearly a way of presenting data in a recency biased environment - you as the analyst greatly intensify the recency bias and lets management feed on it by communicating at this level. Both the analyst and the executive are destined to overreact in an environment like that.
Now go back and apply critical thinking; be Critical, be Empirical and be Rational!
Conclusion
Awareness of the Recency bias is not always enough to stop us from making bad decisions or unfortunate overreactions, but by applying a rule set for critical thinking we provide an opening for us to be cautious in the conclusion we draw.
Side note:
Another really exciting bias (actually somewhat depressing though) is the one brought up by Nassim Nicholas Talib in his books Fooled by Randomness and in particular The Black Swan, where he concludes that we tend to focus on what we know, as opposed to what we do not know.
Cheers
Dennis
Recency Bias in Web Analytics consistently lead to bad decisions and in worst case unfortunate overreactions! Overcome this widespread web analyst error in thought by first and foremost being aware of the bias existing - and install a rule set for critical thinking.
Biases are flaws in the way we reason and it cause us to make errors - and it is a verified fact that we place a disproportionate value on recent events (the Recency Bias) than to those events at the beginning or the middle of an observation. You know this in simple scenarios such as feeling really good about email as an online campaign channel, two days after the quarterly newsletter was sent out, which of course is a dreadfully biased feeling. This is not just a comical thinking defect, but a serious problem as the days of winging online marketing campaigns with great return on investments are long gone. The only way to win in online marketing today, being superior in comparison to your competitors, is to outthink them.
All that said, in a headline hungry society, people in general exacerbates the downside prospective of the recency bias, but it is even harder to beat this bias when we are presented with the positive effect of recency around the Internet as in e.g. Google Organic Ranking, Technorati Ranking etc. Therefore, more than ever, we should stress the importance of applying a rule set for critical thinking when working to overcome the Recency Bias.
Rule set for critical thinking
– overcoming the Recency Bias.
- Be critical
- Be empirical
- Be rational
Be critical
Be very critical towards your own conclusions, and this might sound undemanding and effortless, but it actually requires extensive experience in identifying the extent of one’s own ignorance. You become a whole lot less biased if you are intellectually humble – and if this is not within your character, practice this when you are by yourself. :-)
Note:
A great exercise is recalling previous beliefs that you once held strongly (and we all have that), but now reject and at best almost find embarrassingly naive. If you are out of suggestions, think about how you did online campaign attribution five years ago or how you did web site testing 5 years ago.
Be empirical
Let the data (the experiment itself) guide you, not the hypothesis you set to begin with. It is extremely important that you experiment where you do not know and that you make sure to read the numbers for what they are – we are all eager to force a complex situation (reality) into our somewhat simplified models.
Note:
A great way of building an empirical attitude is to simply ask your visitors what they think about a given hypothesis – it will also, quite interestingly conclude whether it was a question worth asking to begin with.
Be rational
Rationality and reason should be our key methods used to analyze any data set. And as a comment to that - the recency bias can result from an excess of information, disabling rational thought, where the solution should be to eliminate data that is not needed to conclude or cannot be acted upon. Information that cannot be acted upon simply distracts and should be avoided at any point. This is of course a general rule of thumb in Web Analytics.
Note:
Applying context is a fair initiative in being rational (not acting in a biased manner). And a great way to reduce the Recency Bias applying context is through data visualization like Sparklines - which you saw introduced with e.g. Google Analytics in the latest version. I wonder if this was to reduce the Recency Bias though – perhaps Avinash can elaborate on that when we meet next. :-)
Example on Recency Bias in Web Analytics
I honestly believe we on a regular basis are biases in our decision making, I also believe that we are unfairly presented with results in a biased environment, making it even harder to overcome. And a typical example of the Recency Bias in action is what we see in the use of Dashboards. Where my objection to most executive dashboards is that they tend to show only current values of a few metrics, taken completely out of context, and with little or no history applied to them.
Take the following Gauge (which to begin with, is a poor dashboard visualization form, in anything but real-time environments):
25% Paid Search visit to sale Conversion Rate (bear in mind, that this is in fact a real Dashboard item from a client of ours). To me - this sounds pretty damn fantastic – and it is presented in green and all, so it must be “good”. So what am I, as an executive, supposed to think now? Other than Paid Search is it!
Beyond the obvious – that this is clearly a way of presenting data in a recency biased environment - you as the analyst greatly intensify the recency bias and lets management feed on it by communicating at this level. Both the analyst and the executive are destined to overreact in an environment like that.
Now go back and apply critical thinking; be Critical, be Empirical and be Rational!
Conclusion
Awareness of the Recency bias is not always enough to stop us from making bad decisions or unfortunate overreactions, but by applying a rule set for critical thinking we provide an opening for us to be cautious in the conclusion we draw.
Side note:
Another really exciting bias (actually somewhat depressing though) is the one brought up by Nassim Nicholas Talib in his books Fooled by Randomness and in particular The Black Swan, where he concludes that we tend to focus on what we know, as opposed to what we do not know.
Cheers
Dennis
Labels: Avinash, Critical, Empirical, Google Analytics, Rational, Recency Bias, Web Analytics
Posted at: 20:35 
4 Comments
Post a comment
Subscribe
