In qualitative analysis 5-8 users are enough

First; if you do not read Robbins conversion blog – add it to your list, it belongs there. Secondly, there was a great post the other day: Conversion on a limited budget: Users vs experts – which you should read. I had a reply to the post or more precisely the one comment saying: “I highly doubt that I am going to learn anything with 5-8 users”.

I honestly think that this is incorrect. If one read studies like: “A mathematical model of the finding of usability problems” and other similar literature like “Stochastic Processes” – one will see that a very limited number of users have a high impact in most user tests, whether that be for better usability and or specifically for increased conversion; which by the way typically is the result of increased usability.

I thought it would be appropriate to extend that comment with a bit of visualization to back it up. :-)

Jacob Nielsen introduced a model some time ago saying:

N(1-(1-L) to the power n)

• N is the total number of usability problems in the given design
• L is the proportion of problems discovered testing 1 user (he also guides about a typical value of L at 31% based on their studies)

Let us try to “compute” that in excel so that we can create a graph:
(File: percentage-of-problems-found-with-n-users.xls)



Using and X Y scatter chart we will get a graph that by some means concludes, first of all the diminishing value of users beyond eight users in a test and in particular that adding more than ten users to a test is very hard to justify.



Thus I think we confidently can say that one WILL discover (learn) most problems by using only 5-8 users. This is of course based upon the fact that we believe in Jacob’s Model. I do! (being Danish and all.. :-)

Measuring dynamic internal search performance

As a follow-up to my previous post on how to use Web Analytics to determine the width of your Internal Search Query box – I thought it would be suitable for me to deploy an internal search utility on my blog. Looking around for a solution (beyond just adding a link to technorati) that could provide us with quick, accurate and measurable results – I ended up deciding to use the Google AJAX Search API. First, because it is free (no money is made on this blog), second because I could “code” it to my taste... and it is actually enjoyable being geeky from time to time :-)



I caught myself in trying to code the search utility for optimized tracking and not for optimized usability – this, as in I wanted the “normal” Search Query Box functionally:

1. type in full query
2. refresh screen with a set of results (where the search query and results kpi’s are tracked)

However; Internal Search utilities are changing and I finally decided to go with the potentially better “Web 2.0” solution (time will tell). A solution where instant results are shown on an unchanged page after each keystroke.

Let us try to assess the results before talking about WHAT to measure: Searching for e.g. “emetrics san francisco 2007” we get:



Search results - “Web 1.0”
1. Emetrics San Francisco 2007
2. My comments to Emetrics London 2007
3. NEW Google Analytics launched at Emetrics SF Today!
4. Notes from ad:tech San Francisco
5. 9 Ways to Make Money on Analytics

Setting up a similar and comparable list is simply not possible in a dynamic and instant web 2.0 results environment (as the one deployed on this blog) – as we have instant results after each keystroke. We can on the other hand look at the numbers of results per keystroke:

Search results - “Web 2.0”
1 results for the phrase “e”
0 results for the phrase “em”
0 results for the phrase “eme”
0 results for the phrase “emet”
0 results for the phrase “emetr”
0 results for the phrase “emetri”
0 results for the phrase “emetric”
8 results for the phrase “emetrics”
8 results for the phrase “emetrics ”
5 results for the phrase “emetrics s”
0 results for the phrase “emetrics sa”
3 results for the phrase “emetrics san”
5 results for the phrase “emetrics san ”
3 results for the phrase “emetrics san f”
0 results for the phrase “emetrics san fr”
0 results for the phrase “emetrics san fra”
0 results for the phrase “emetrics san fran”
0 results for the phrase “emetrics san franc”
0 results for the phrase “emetrics san franci”
3 results for the phrase “emetrics san francis”
3 results for the phrase “emetrics san francisc”
5 results for the phrase “emetrics san francisco”
5 results for the phrase “emetrics san francisco ”
0 results for the phrase “emetrics san francisco 2”
0 results for the phrase “emetrics san francisco 20”
0 results for the phrase “emetrics san francisco 200”
5 results for the phrase “emetrics san francisco 2007”

So dependent on how quick I am to type and how quick my internal search utility is to respond I could end up performing 27 searches. If I am semi fast in typing in the query I might end up doing a random 10 or 15 searches, furthermore I might not have the qualification to look at the screen while I type and then as a usability fact “only” see the final result. Another interesting fact is that, if I were in fact searching for “Emetrics San Francisco 2007“ and that I would type and see results at the same time, I would stop after “emetrics”, but even more challenging, this would only count as a success search phrase keyword until I write a new post about Emetrics which would then be on top and then the “Emetrics” term might not be enough to guide people to the “emetrics San Francisco 2007“ post.

This is really fascinating!? :-)

Let us try to have a look at the standard KPI’s (a off the top of my head list - so do not take this as a final list of the most used Internal Search KPI’s, however; the list still rings true and must still pretty much be expected of anybody doing decent internal search tracking)

KPI’s for Internal Search - “Web 1.0”

• Search action (whether a search have submitted)
• Search phrase
• The number of search results
• No results action
• The number of searches per visit/visitor
• Search successful (as in clicking on a result)
• The number of search attempts before success
• Advanced search action (narrowing down the search)
• Search location

KPI’s for Internal Search - “Web 2.0”

• ?

I have plenty of ideas, as I think it is somehow obvious that the standard static internal search Web 1.0 KPI’s are not very suitable for a Web 2.0 environment, but I have no data as we speak, so it is time to deploy (from a tracking point of view) a few of them and see how actionable the results are. And this might even give me an opportunity to consult our Oracle “Avinash” or the smart people at OX2 as well :-) - More about the subject later when I have had some time to do a bit of research. And PLEASE do leave a comment with your ideas. :-)

N.B.
If you want great input for how to evaluate static internal search performance – go look at Gary Angel’s input on the matter. It is as usual a first-class post.

Use Web Analytics to determine the width of your Internal Search Query box

This is fairly mundane task that all of us should do once in a while – making sure that one of the most important site utilities are optimized for optimal user experience (usability) and thus in the end optimal site success; as in MORE money! And the search utility can easily be concluded being among the most important tools by running a basic “Revenue Participation Report” - should you be in doubt whether it is worth doing the task.



Looking at the above report outcome from a random British retailer we instantly see that the search utility participates in £51,416.62 of the overall £120,529.88 revenue for the given period – a whopping 43%. I think we can conclude that for this retailer, where almost half the revenue is involved with site search, it is of the utmost importance that one of the few important design parameters, such as the search query box width, is “calculated” and not just spinelessly handed over to the web-designer as a last minute one man decision.

I find it difficult to persuade myself that a reasonable A/B and or Multivariate testing can be applied with success to this task, given the restrains of different user behaviour when changing this parameter – as in the observable fact that users use longer queries when the search box is wider. And I think that we in general just have to accept that we can get other insights when using qualitative studies.

So; let us have a look at the immediate available data (the number of Searches per Search Query):



Looking at the distribution of this dataset, it looks like a standard long tail (this is a completely different dialogue, but beware of a drooping tail) – However; that is not why we are here today. What we need to do is conclude on: how many characters is needed to accommodate a certain percentage of all search queries in full. I know this is obvious, anyway – now you will at least know how I went about it in EXCEL:

1.
Do a volume export of the upper report (Search Phrase with the number of searches for that phrase)

2.
Import this dataset into EXCEL (in the upper example we have just about 24000 different Search Phrases in the given period)

3.
Add a third column called e.g. “Lenght” – which is the number of characters in that particular search phrase (EXCEL command: LEN() – See picture below, which of course resembles the one from IndexTools’s reporting system)



4.
Then group all those search phrases (rows) with similar length, so that we get a total SUM of searches per search length (Do a quick PivotTable with length being a row – See picture below, which should when summed up still have the same number of total searches, some 24000)




5a.
Then add a column which calculates a “Percent of all Search queries”, so we can start to see how many queries we can accommodate in a search box width of n length. (looking at the picture below - out of the row based data we can see that with as little as 16 Characters we will be able to accommodate full usability on 71% of all our internal search queries).





5b.
Sometimes it is a bit easier to look at this in a scatter chart - for a full view of this see the picture below:




At this time we have all the data needed to conclude on the character width of our internal search box and you might by now ask, why is it that we do not just deploy a search box of e.g. 55 characters width like Google? – But with well researched usability guidelines recommending the internal Search Box being on every page, one simply have to converse space. Therefore in the end, having to make a trade-off, we have to decide how many users we want to fully accommodate by providing them the space needed for their query. The research I have read and agree width concludes that a trade-off (based on the distribution) around 90% of all internal search queries should fit in your search box.

So with that said the conclusion for the site debated above is that:
Internal Search Query box WIDTH should be 23 characters.

Remember; there is no magic number for you to use as this is very much site specific and you simply have to do your own analysis - sorry! :-). I personally recommend redoing this every year.

N.B.
The site debatted have an internal search box width on 35 Characters today and could therefore “save” 12 characters and by that free up space for increased usability. (there is of course other problems with the setup of this one, but that is for another dialogue)