The scope of the study (the tasks and questions)
The scope is a difficult subject to tackle because stakeholders very often want to use a usability study to cover everything at once, and the UX researcher will have to scale the expectations down to a feasible scope. The scope is constrained by what can reasonably be asked of the participants within the timeframe of the study. A video-based remote usability study will have a different test duration than a survey-based one, for example. The budget also influences the timeframe, and subsequently, the scope.
The goal will determine whether the study's scope should comprise the entire product, only certain functions of it, or even just individual screens. If the goal is to determine the usability status quo of a product, for example, then the scope will encompass the entire product, whereas comparing three design alternatives for a new feature will be restricted to that feature.
The goal will also help to identify which tasks are candidates for the study. A goal of increasing conversion during checkout, for example, will necessarily require that at least one task covers the checkout function. Other tasks may include searching for a suitable product to buy or managing the shopping cart.
The UX researcher should list all of the tasks with an estimated duration and the success criteria for each task. Often, the average task duration can be provided by the stakeholders. Or, the UX researcher may suggest a ballpark. Each task duration can be a rough estimate because there is a lot of variance in the time actually required by study participants to complete a task. The success criteria describe what has to be achieved in order to consider a task successfully completed. For example, if the study participants are testing a checkout journey, the success criteria might be that they complete their purchase, or that they reach the entry of payment information.
If the list of tasks is too extensive, or if the duration is too long, the UX researcher and the study sponsor have to prioritize them, in order to create a shortlist. The goals will help to determine how to prioritize the tasks. For example, tasks in a competitive analysis should comprise the core tasks of the product – the ones that are the main reasons for users using the product. Other ways to prioritize the tasks include sorting them according to frequency of use, or focusing on newly added features or critical tasks that, when done incorrectly, might have severe consequences. Essential tasks, such as logging in or registering, must be a part of the study if they are a prerequisite to using the product under testing. Auxiliary tasks, such as setting preferences, are of secondary priority.
The UX researcher should inform the stakeholders of which benchmark scores they are planning to use, if any, and ask if the study sponsor has any requirements. The benchmark scores increase the time required by a participant completing the study, and should therefore also be listed as a part of the scope. Larger companies might have customized benchmark scores that they use, in order to standardize the results and make them comparable. If this is the case, the UX researcher must understand which questions to ask, how to analyze the responses, and how to derive the score. Examples of some benchmark scores that can be used are covered in the following sections.