Designing usability tests around comprehension to catch designs that are easy to use but misunderstood

Why I run usability tests

Usability testing isn’t only about safeguarding the quality of a product’s experience at the interaction level. Looking back on running tests within agile teams, I believe it brings three kinds of value to the product, the team, and project management:

My testing strategy

According to ISO 9241-11, the three core components of usability are effectiveness, efficiency, and satisfaction. This is a comprehensive, widely recognized framework. In my own testing practice, however, I’ve found that putting more emphasis on checking participants’ comprehension tends to produce insights that get right to the heart of what I need to understand. This adjustment comes mainly from two observations:

Comprehension lends itself to more precise questions than satisfaction does
- Satisfaction focuses on participants’ emotional reactions, but when they’re being observed and under time pressure, participants often struggle to explain on the spot why they feel the way they do. Questions designed around comprehension (for example, asking participants to explain in their own words what they just did) let me pinpoint more directly where a gap in understanding occurs.
Participants’ cultural context: Japanese etiquette
- Most of my participants were Japanese users. In Japanese culture, people tend to be mindful of etiquette and avoid voicing negative opinions directly to strangers. Even when I emphasized before the test that they could speak frankly, participants rarely expressed dissatisfaction on their own. This made satisfaction questions less informative in my actual testing and led me to rely more on comprehension.

My question framework for comprehension

Understanding of the action: Can participants clearly state what they just did and why?
Understanding of the interface: Do participants truly understand the UI elements and wording on the screen (such as labels and copy)?
Understanding of the context: In the given task scenario, do participants understand why they need to complete the task?

As long as we objectively record effectiveness and efficiency during the test and verify comprehension through questions, we can pinpoint where participants run into frustration or comprehension issues and use those insights to improve the design. Once users understand the product and can use it efficiently, satisfaction after launch follows naturally.

When I run usability tests

What triggers a test?

Usability testing takes time and people, so not every feature gets one. Some features have a broad scope and a complex user flow, and we need to verify that users form the right expectations and mental model of how the system works. For features like these, I discuss with the team whether to run a usability test.

My testing process

My testing process has three phases: (1) preparation, (2) execution, and (3) analysis and follow-through.

🗂 Preparation (steps 1–5)

1. Define the test goal

Although the purpose and core evaluation criteria of usability testing are already clear, I still use this step to describe what it looks like when users use the feature effectively and smoothly, so that the team shares the same understanding of the test. This becomes the evaluation baseline for the whole test.

2. Break down the test goal

I break the high-level test goal down into specific sub-goals that can be assessed. In doing so, I follow the perspective described above: effectiveness and efficiency correspond to participants’ behavior, while comprehension corresponds to their cognition. These sub-goals then form the core structure of the test script.

3. Write the test script

Design tasks
- I turn the sub-goals into concrete test tasks. Each task includes a scenario and an instruction. The scenario provides context and helps participants step into the mindset of a real user; once the scenario is established, a clear instruction prompts them to start using the prototype.
Set success criteria for each task
- I clearly define what counts as success in terms of behavior. Because a task can often be completed in more than one way, the success criteria should cover as many valid paths as possible and take differences in efficiency between paths into account.
Design comprehension questions for after each task
- For key tasks where correct understanding is essential, I prepare follow-up questions in advance. Even if participants show no problems with effectiveness or efficiency, I use these questions to check whether they truly understand the meaning behind what they did.

4. Run a pilot test

A finished test script inevitably has blind spots, so before the actual test, I debug it with an internal pilot test. I usually check four aspects:

5. Recruit participants

When recruiting participants for usability tests, I usually take a practical and flexible screening approach:

Ideally, participants should match the target users
- I set screening criteria based on the target users of the feature or product, so that participants’ behavior reflects how target users actually use it.
Practical trade-offs and minimum requirements
- In practice, participants who match the target user profile are scarce, so I usually save these valuable participants for concept tests or generative interviews, which require digging deep into values and needs.
- Usability testing, by contrast, focuses on checking interface logic and information architecture, so I set the minimum requirement at “participants have basic skills in using everyday digital devices.” This rules out false usability issues caused by unfamiliarity with digital devices, so the test can focus on real design blind spots.

🎬 Execution (step 6)

6. Run the usability test

Test environment
- Depending on the situation, tests are run remotely or in person. For remote tests, I ask participants to turn on their camera and share their screen. In-person tests take place in a company meeting room, with a camera set up on a tripod to fully record participants’ screens, facial expressions, and body language.
Two-person observation setup
- To ensure the quality of both the session and the notes, each session is split between two roles:
  - Facilitator: Runs the test according to the script, explains the tasks, and asks follow-up questions as needed.
  - Observer: Stays in the background, focusing on participants’ facial expressions, body language, actions, and comments, and takes detailed notes.
Getting the team involved
- I encourage the product team (POs, engineers, and others) to take part in testing. Team members can sign up as observers to see users’ pain points firsthand, which greatly reduces communication costs when the design needs to change later. If team members want to try being the facilitator, I share the test script in advance and help them prepare, which gradually raises the whole team’s design maturity.

📊 Analysis and follow-through (steps 7–8)

7. Analyze the results

Visualizing qualitative data and evaluating tasks
- I break the session transcripts into single-idea units and put each one on a sticky note in Miro. I then build a participant-by-task analysis grid in Miro (participants across the top, tasks down the side) and place each sticky note in the matching cell.
Evaluating task results as a team
- I facilitate a team review of each cell, and we rate each result against the following criteria:
  - 🟢 Completed: The participant completed the task with no notable efficiency or comprehension issues.
  - ⚠️ Completed with issues: The participant completed the task but had notable efficiency or comprehension issues along the way.
  - ❌ Not completed: The participant gave up or couldn’t achieve the task goal.
Analyzing and defining issues
- Severity scoring
  - The team and I score each cell (⚠️ = 0.5 points, ❌ = 1 point) and total the scores for each task. The totals show which tasks have high severity, and we analyze the usability issues in the highest-scoring tasks first.
- Affinity diagramming
  - I pick out the key sticky notes that point to the causes of usability issues and synthesize them across participants, consolidating scattered pain points into clearly defined usability issues.
- Prioritizing fixes
  - I facilitate a team discussion of the defined usability issues, weighing fix effort against issue severity, and we decide together on the order of fixes.

8. Iterate on the design and hand off to development

Based on the fix priorities set in the previous step, I turn test insights into specific design proposals and confirm with the team which changes go into the current development cycle and which go into the backlog.

Example: the learning plan feature—a comprehension issue that didn’t stop anyone from completing the task but still had to be fixed first

In testing the learning plan feature’s onboarding, 60% of participants misread the “weekly study time” they had set as “daily” study time.

Although participants had no effectiveness or efficiency problems, I argued to the team that this issue had to be fixed first, because it was an error in comprehension. It wouldn’t cause trouble right away, but it would surface later, when users started following their plan, as confusion along the lines of “Wait, this isn’t what I expected.” This is also why I focus on comprehension in testing: some usability issues don’t show up in how smoothly people use the product in the moment. The team members who had joined as observers had seen the misunderstanding firsthand, which is why they accepted this priority.

The cause turned out to be more fundamental than I had thought.

When I first designed this screen, I assumed that a clearly written section heading (“Please select your weekly study time”) would be enough for users to understand that the time was weekly. But testing showed me how users actually behave: users don’t read even the headings word for word; they move quickly through the screen based on their own assumptions. When the screen showed only a row of duration options such as “30 min, 1 hour…,” users saw a length of time and instinctively read it as time per day, overlooking the heading—and with it, the key premise of “weekly.”

My solution: moving the design intent from copy to interaction.

I split out “How many days a week will you study?” as its own question that users had to answer themselves, placed before “How long will each session be?” This way, the weekly unit was no longer an explanation users could skip but a step they couldn’t avoid; the design intent was built into the flow itself. This added a step, but in the retest, participants no longer misread weekly as daily.