The number of participants
Moderated studies are usually qualitative studies, because running a moderated, quantitative study (requiring a larger number of participants) is very cost-intensive. Unmoderated studies can be both quantitative or qualitative, due to their scalability.
We are focusing only on remote, qualitative studies in this book, and for moderated studies, we recommend the following:
- 3-5 participants if you're on a budget or tight deadline, or if you're going to run a series of studies during your development cycle.
- The next step up is 6-8 participants, mostly because you can fit that in a (full) day. You can plan for 8 participants but stop at 5 or 6 if you see that you're not getting any additional feedback.
- 12-15 participants (or more), if the budget and schedule allow it or if more features need to be tested than would fit into a regular session.
For unmoderated studies, we recommend the following:
- 8-12 participants for video-based studies.
- For survey-based studies, larger numbers, such as 15-25 participants, work well, as there is less effort involved in evaluating survey responses, as compared to video feedback.
- For hybrid studies (video- and survey-based), we recommend 12-15 participants.
All figures are per target user group and are merely guidelines.
Even though we have laid it out so cleanly, it is a very contentious topic. Often, a stakeholder may request that a study is executed with more participants, because "more is better, right?" We then defer to Nielsen's study - "After the fifth user, you are wasting your time by observing the same findings repeatedly but not learning much new" (https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/):
If they cannot be convinced, we capitulate. There is no harm in running a larger study, other than the increased cost and effort.
The product development methodology may also dictate the number of participants to invite. Agile methodologies advocate a faster, more iterative process, and recommend using as few as 3-5 participants for both moderated and unmoderated studies.