27

May

By Dennis R. Mortensen
3 comments
In qualitative analysis 5-8 users are enough
First; if you do not read Robbins conversion blog – add it to your list, it belongs there. Secondly, there was a great post the other day: Conversion on a limited budget: Users vs experts – which you should read. I had a reply to the post or more precisely the one comment saying: “I highly doubt that I am going to learn anything with 5-8 users”.

I honestly think that this is incorrect. If one read studies like: “A mathematical model of the finding of usability problems” and other similar literature like “Stochastic Processes” – one will see that a very limited number of users have a high impact in most user tests, whether that be for better usability and or specifically for increased conversion; which by the way typically is the result of increased usability.

I thought it would be appropriate to extend that comment with a bit of visualization to back it up. :-)

Jacob Nielsen introduced a model some time ago saying:

N(1-(1-L) to the power n)

  • N is the total number of usability problems in the given design
  • L is the proportion of problems discovered testing 1 user (he also guides about a typical value of L at 31% based on their studies)
Let us try to “compute” that in excel so that we can create a graph:
(File: percentage-of-problems-found-with-n-users.xls)

Using and X Y scatter chart we will get a graph that by some means concludes, first of all the diminishing value of users beyond eight users in a test and in particular that adding more than ten users to a test is very hard to justify.

Thus I think we confidently can say that one WILL discover (learn) most problems by using only 5-8 users. This is of course based upon the fact that we believe in Jacob’s Model. I do! (being Danish and all.. :-)

  • Avinash Kaushik

    I humbly apologize but I am going to have to disagree with the Danes.

    The above outlined thought process works powerfully for widget or software uses. It kinda sorta breaks on the web a little bit.

    Most often now websites are being used for purposes well beyond those intended by the creator/owner. And that is not such a bad thing.

    On any commercial website (for profit or non profit) if you ask Visitors for their Primary Purpose (“why are you here on our website”) you’ll get a diverse set of answers (even on a hard core ecommerce website).

    So the problem with Jakob – Mortensen proposal as applied to the web is that if you go find 5 – 8 people then it is likely that you might, with lots of good luck, only find one representing each Primary Purpose. Would you trust one person’s opinion?

    If Visitors come for six or eight main Primary Purposes, would you conduct above analysis approx 50 people to identify all the problems?

    This is the reason I am such a fan of moving beyond traditional qualitative methodologies into some of the emerging ones like a/b or multivariate testing, remote usability testing (with live recruiting not let’s find people and remotely test), etc. Your chances of learning from real people in real situations increases exponentially.

    As a complement to these newer methodologies the traditional methodologies can help identify big issues, especially in early prototype stages.

    I welcome your feedback.

    -Avinash.
    PS: As soon as I hit submit on this comment I am going to hide behind my desk to protect myself from having incurred the wrath of the Danes!! :)

  • Dennis R. Mortensen

    Response to Avinash.

    Thank you very much for your comments Avinash – enlightened and thought through as always! However; I still believe in my post and I find the model introduced by Jacob very much valid. :-)

    I am sure that we are not that far apart at the end of the day and it might even be my broken Danish/English – but I would like point out the following:

    If one run an A/B and or Multivariate test, it is expectantly with a specific optimization purpose in mind, otherwise one cannot measure whether success have been achieved – and in general, optimization when looking at these “emerging technologies”, is about increased Conversion, but it could be any measurable success action. I think we all agree on this.

    So when saying that there is a potential danger in the numbers of users needed for qualitative analysis – model, by questioning the “quality” of the users, with the fact that they might have multiple purposes. It is unjust to compare a MVT (multivariate test) with a narrow and specific purpose in mind to a complete website qualitative analysis (as you indicate).

    Therefore one should when running a usability study (or other qualitative analysis) have a specific purpose in mind – and that purpose should be communicated to the users. This can be as wide as in e.g. “Go to our website and find product x, buy it and have it sent to your home address” – but even with this wide set of actions, the users have a similar purpose. AND having 5-8 users with that purpose in mind you will find “most” problems on their way there.

    So it is probably me, not being clear to begin with. I am not saying that it takes 5 users in 1 test to test the whole of your website. I am however saying that it takes no more than 5-8 users to run a valid and comprehensive qualitative test with a specific purpose in mind. I do agree that you have to run multiple test. I actually believe that one have to continuously run testing, both qualitative and quantitative – and I definitely believe highly in these emerging technologies.

    Hitting the submit button: ….And if I remember my history lessons correct, I think the furthest the Vikings (Danes) ever got was Greenland, North America, but not California though. But you might see me in my Speedos at SES San Jose.. so watch out my friend – raid and pillage among the other analytics exhibitors is not a pretty sight :-)

  • Dennis R. Mortensen

    Note:
    Check out the comments from Robbin on “How significant is user testing?” ..

    I suppose the blogger trackback should have picked it up…