In qualitative analysis 5-8 users are enough

First; if you do not read Robbins conversion blog – add it to your list, it belongs there. Secondly, there was a great post the other day: Conversion on a limited budget: Users vs experts – which you should read. I had a reply to the post or more precisely the one comment saying: “I highly doubt that I am going to learn anything with 5-8 users”.

I honestly think that this is incorrect. If one read studies like: “A mathematical model of the finding of usability problems” and other similar literature like “Stochastic Processes” – one will see that a very limited number of users have a high impact in most user tests, whether that be for better usability and or specifically for increased conversion; which by the way typically is the result of increased usability.

I thought it would be appropriate to extend that comment with a bit of visualization to back it up. :-)

Jacob Nielsen introduced a model some time ago saying:

N(1-(1-L) to the power n)

• N is the total number of usability problems in the given design
• L is the proportion of problems discovered testing 1 user (he also guides about a typical value of L at 31% based on their studies)

Let us try to “compute” that in excel so that we can create a graph:
(File: percentage-of-problems-found-with-n-users.xls)



Using and X Y scatter chart we will get a graph that by some means concludes, first of all the diminishing value of users beyond eight users in a test and in particular that adding more than ten users to a test is very hard to justify.



Thus I think we confidently can say that one WILL discover (learn) most problems by using only 5-8 users. This is of course based upon the fact that we believe in Jacob’s Model. I do! (being Danish and all.. :-)

Measuring dynamic internal search performance

As a follow-up to my previous post on how to use Web Analytics to determine the width of your Internal Search Query box – I thought it would be suitable for me to deploy an internal search utility on my blog. Looking around for a solution (beyond just adding a link to technorati) that could provide us with quick, accurate and measurable results – I ended up deciding to use the Google AJAX Search API. First, because it is free (no money is made on this blog), second because I could “code” it to my taste... and it is actually enjoyable being geeky from time to time :-)



I caught myself in trying to code the search utility for optimized tracking and not for optimized usability – this, as in I wanted the “normal” Search Query Box functionally:

1. type in full query
2. refresh screen with a set of results (where the search query and results kpi’s are tracked)

However; Internal Search utilities are changing and I finally decided to go with the potentially better “Web 2.0” solution (time will tell). A solution where instant results are shown on an unchanged page after each keystroke.

Let us try to assess the results before talking about WHAT to measure: Searching for e.g. “emetrics san francisco 2007” we get:



Search results - “Web 1.0”
1. Emetrics San Francisco 2007
2. My comments to Emetrics London 2007
3. NEW Google Analytics launched at Emetrics SF Today!
4. Notes from ad:tech San Francisco
5. 9 Ways to Make Money on Analytics

Setting up a similar and comparable list is simply not possible in a dynamic and instant web 2.0 results environment (as the one deployed on this blog) – as we have instant results after each keystroke. We can on the other hand look at the numbers of results per keystroke:

Search results - “Web 2.0”
1 results for the phrase “e”
0 results for the phrase “em”
0 results for the phrase “eme”
0 results for the phrase “emet”
0 results for the phrase “emetr”
0 results for the phrase “emetri”
0 results for the phrase “emetric”
8 results for the phrase “emetrics”
8 results for the phrase “emetrics ”
5 results for the phrase “emetrics s”
0 results for the phrase “emetrics sa”
3 results for the phrase “emetrics san”
5 results for the phrase “emetrics san ”
3 results for the phrase “emetrics san f”
0 results for the phrase “emetrics san fr”
0 results for the phrase “emetrics san fra”
0 results for the phrase “emetrics san fran”
0 results for the phrase “emetrics san franc”
0 results for the phrase “emetrics san franci”
3 results for the phrase “emetrics san francis”
3 results for the phrase “emetrics san francisc”
5 results for the phrase “emetrics san francisco”
5 results for the phrase “emetrics san francisco ”
0 results for the phrase “emetrics san francisco 2”
0 results for the phrase “emetrics san francisco 20”
0 results for the phrase “emetrics san francisco 200”
5 results for the phrase “emetrics san francisco 2007”

So dependent on how quick I am to type and how quick my internal search utility is to respond I could end up performing 27 searches. If I am semi fast in typing in the query I might end up doing a random 10 or 15 searches, furthermore I might not have the qualification to look at the screen while I type and then as a usability fact “only” see the final result. Another interesting fact is that, if I were in fact searching for “Emetrics San Francisco 2007“ and that I would type and see results at the same time, I would stop after “emetrics”, but even more challenging, this would only count as a success search phrase keyword until I write a new post about Emetrics which would then be on top and then the “Emetrics” term might not be enough to guide people to the “emetrics San Francisco 2007“ post.

This is really fascinating!? :-)

Let us try to have a look at the standard KPI’s (a off the top of my head list - so do not take this as a final list of the most used Internal Search KPI’s, however; the list still rings true and must still pretty much be expected of anybody doing decent internal search tracking)

KPI’s for Internal Search - “Web 1.0”

• Search action (whether a search have submitted)
• Search phrase
• The number of search results
• No results action
• The number of searches per visit/visitor
• Search successful (as in clicking on a result)
• The number of search attempts before success
• Advanced search action (narrowing down the search)
• Search location

KPI’s for Internal Search - “Web 2.0”

• ?

I have plenty of ideas, as I think it is somehow obvious that the standard static internal search Web 1.0 KPI’s are not very suitable for a Web 2.0 environment, but I have no data as we speak, so it is time to deploy (from a tracking point of view) a few of them and see how actionable the results are. And this might even give me an opportunity to consult our Oracle “Avinash” or the smart people at OX2 as well :-) - More about the subject later when I have had some time to do a bit of research. And PLEASE do leave a comment with your ideas. :-)

N.B.
If you want great input for how to evaluate static internal search performance – go look at Gary Angel’s input on the matter. It is as usual a first-class post.

Use Web Analytics to determine the width of your Internal Search Query box

This is fairly mundane task that all of us should do once in a while – making sure that one of the most important site utilities are optimized for optimal user experience (usability) and thus in the end optimal site success; as in MORE money! And the search utility can easily be concluded being among the most important tools by running a basic “Revenue Participation Report” - should you be in doubt whether it is worth doing the task.



Looking at the above report outcome from a random British retailer we instantly see that the search utility participates in £51,416.62 of the overall £120,529.88 revenue for the given period – a whopping 43%. I think we can conclude that for this retailer, where almost half the revenue is involved with site search, it is of the utmost importance that one of the few important design parameters, such as the search query box width, is “calculated” and not just spinelessly handed over to the web-designer as a last minute one man decision.

I find it difficult to persuade myself that a reasonable A/B and or Multivariate testing can be applied with success to this task, given the restrains of different user behaviour when changing this parameter – as in the observable fact that users use longer queries when the search box is wider. And I think that we in general just have to accept that we can get other insights when using qualitative studies.

So; let us have a look at the immediate available data (the number of Searches per Search Query):



Looking at the distribution of this dataset, it looks like a standard long tail (this is a completely different dialogue, but beware of a drooping tail) – However; that is not why we are here today. What we need to do is conclude on: how many characters is needed to accommodate a certain percentage of all search queries in full. I know this is obvious, anyway – now you will at least know how I went about it in EXCEL:

1.
Do a volume export of the upper report (Search Phrase with the number of searches for that phrase)

2.
Import this dataset into EXCEL (in the upper example we have just about 24000 different Search Phrases in the given period)

3.
Add a third column called e.g. “Lenght” – which is the number of characters in that particular search phrase (EXCEL command: LEN() – See picture below, which of course resembles the one from IndexTools’s reporting system)



4.
Then group all those search phrases (rows) with similar length, so that we get a total SUM of searches per search length (Do a quick PivotTable with length being a row – See picture below, which should when summed up still have the same number of total searches, some 24000)




5a.
Then add a column which calculates a “Percent of all Search queries”, so we can start to see how many queries we can accommodate in a search box width of n length. (looking at the picture below - out of the row based data we can see that with as little as 16 Characters we will be able to accommodate full usability on 71% of all our internal search queries).





5b.
Sometimes it is a bit easier to look at this in a scatter chart - for a full view of this see the picture below:




At this time we have all the data needed to conclude on the character width of our internal search box and you might by now ask, why is it that we do not just deploy a search box of e.g. 55 characters width like Google? – But with well researched usability guidelines recommending the internal Search Box being on every page, one simply have to converse space. Therefore in the end, having to make a trade-off, we have to decide how many users we want to fully accommodate by providing them the space needed for their query. The research I have read and agree width concludes that a trade-off (based on the distribution) around 90% of all internal search queries should fit in your search box.

So with that said the conclusion for the site debated above is that:
Internal Search Query box WIDTH should be 23 characters.

Remember; there is no magic number for you to use as this is very much site specific and you simply have to do your own analysis - sorry! :-). I personally recommend redoing this every year.

N.B.
The site debatted have an internal search box width on 35 Characters today and could therefore “save” 12 characters and by that free up space for increased usability. (there is of course other problems with the setup of this one, but that is for another dialogue)

Emetrics San Francisco 2007

I returned safely yesterday, from a great Emetrics San Francisco summit! – No need for me to give you an update; as I am most sure we meet there already :-)
However; you can find some comments from my fellow bloggers here (NB: highly inflated with GA2 posts, but that is only fair, as it is a cool launch):

• Google Analytics Is Re-Launched: Do These Five Things First In V2
• Web Analytics Demystified, Incorporated
• All the news that is fit to Blog
• Emetrics Summit - San Francisco - Pre Day 1
• Emetrics Summit - Day 1 - a long, long day
• Jim Sterne's talk at Emetrics Summit
• Emetrics Summit Wrap up
• eMetrics, web analysts and social behavior
• From the floor of eMetrics
• new google analytics visualizations
• OX2’s involvement following Jim Sterne’s presentation at Emetrics London & San Francisco
• Eric Peterson launches his Company & Aurélie joins his Board of Advisors
• Google Launches Google Analytics V2 at Emetrics San Francisco
• Who’s laughing at Google Analytics now?
• Google Analytics V2: more details
• Four Gurus For You!
• eMetrics Summit

I think a meet every single person who matters in the Web Analytics industry (including most of the above) – and that alone justifies going there. I did a Video Interview with BlastPR and I will surely update this post if I get my hands on it.

NEW Google Analytics launched at Emetrics SF Today!

Today at Emetrics San Francisco 2007 Brett Crosby and Jeffrey Veen from Google announced the NEW Google Analytics – which they indicated have been a year in the works.

It includes among other things updates on:

• Discoverability and data in context
• Customizable dashboards
• Scheduled email reports
• Better visualization
• Improved "segmentation"

Honestly; it looks good!! :-) ... and they will definitely move closer to the "paid for tools" side of the industry. However; it looks like it is still a reporting tool based on the same “bucketed” data collection system. I signed up for the new version for this VisualRevenue blog and you can find the first FRESH screenshots below. I will post some more screenshots when they have collected more data and made it available, which I assume is within the next 12 hours, so stay tuned...

9 Ways to Make Money on Analytics

Eric Enge had a great post over at Search Engine Watch and with a title like this: 9 Ways to Make Money on Analytics - I just had to reference it here. This being a blog about how to increase publisher revenue through analytics! :-)

..And on top of that; I am of course proud that he chose to add my “The Long Tail (drooping tail theory)... and how to calculate missing Revenue” as tip #5:

“Spot a missing long tail: There was a great post recently by Dennis Mortensen about Spotting a missing long tail (another person whose blog you should read on a regular basis). Dennis explains it really well, but the basic idea is that the traffic on your highest volume page should be matched by the traffic on a larger number of lower volume pages on your site. If this is not the case, then you are missing out on an opportunity. Check out Dennis' post for more details on this suggestion.”

Go check out his post.. it is always worth having a look at what Eric has to say.

Cheers... off to Emetrics San Francisco.

Book: Freakonomics

On my way back... from San Francisco to Europe – I had the pleasure (I had 17 hours of travel to kill) to read Freakonomics the book. I know I am way late in commenting this book, but for some reason it did just not end up on my commute-reading-list.

Since most of you reading my blog most likely read this book already – I will spare you the pompous review and simply conclude that:

"Any serious analyst who are eager to move beyond reporting and basic cross-analysis of a few metrics and into matters beyond the low hanging fruits MUST read Freakonomics the book."

You will get two things from this book that are essential:

1. The desire to ask even more interesting questions with your data
2. The desire to use Regression analysis even more

Let us start to investigate.... :-)