Arctera
2025
AI-assisted Usability Evaluation
Redesigning a manual usability evaluation workflow with AI assistance, reducing scoring time from approximately 1 hour per user to 5–10 seconds while keeping expert judgment in the loop.
Overview
Redesigning how usability evaluations happen
PURE (Pragmatic User Experience Rating by Experts) is a structured usability evaluation framework used to assess task difficulty, efficiency, and intuitiveness.
In my team, PURE evaluations were conducted manually by multiple designers reviewing usability-test recordings together. The process provided a structured way to evaluate usability, but it was also highly time-consuming and difficult to scale.
I explored whether AI could take on the repetitive evaluation work while keeping expert judgment at the center of the process.
The result was an AI-assisted evaluation workflow that reduced PURE scoring time from approximately 1 hour to 5–10 seconds per session.
A rigorous process that didn't scale
The existing PURE evaluation process required 3–4 designers to review each usability session together.
A single evaluation took approximately one hour and often involved repeated alignment discussions before the team reached a final score.
As testing volume increased, this created a growing bottleneck:
• Manual scoring consumed significant designer time
• Multiple designers had to review the same session
• Alignment discussions added further time
• Research synthesis and design decisions were slowed down
• The process became difficult to scale as more usability tests were conducted
The challenge was not the evaluation framework itself. The challenge was the amount of repetitive work required to apply it manually to every session.
Opportunity
The evaluation already contained structured signals
Although PURE scoring was manual, the evidence designers relied on was relatively structured.
Each evaluation could draw from:
• Clearly defined tasks and success criteria
• Task completion or failure
• Time taken to complete tasks
• User hesitation or confusion
• Moderator prompts or nudges
• Text transcripts of the usability session
This revealed an opportunity.
If these inputs could be standardized, could AI help interpret them consistently and generate an evaluation that aligned with expert scoring?
That became the central hypothesis behind the project.
Could AI support expert evaluation without replacing it?
If usability-session inputs—tasks, transcripts, completion data, and moderator interventions—were standardized and structured, AI assistance could help generate PURE evaluations comparable to expert scoring while drastically reducing the time required.
The goal was therefore not to automate usability research.
It was to redesign the evaluation workflow so that AI handled the repetitive interpretation and scoring work, while experts remained responsible for judgment and validation.
Solution
An AI-assisted PURE evaluation workflow
I designed an AI-assisted usability evaluation workflow using a Custom GPT.
The workflow takes structured usability-session information as input and produces a PURE evaluation for each task.
INPUT
• Task definitions and success criteria
• Text transcripts of usability sessions
• Time taken per task
• Moderator prompts or nudges
OUTPUT
• PURE rating for each task
• Qualitative justification for each score
• Indicators of friction, confusion, or inefficiency
The workflow was intentionally designed as decision support rather than a replacement for expert evaluation.
Designed as decision support, not replacement
The workflow was intentionally designed to support the researcher rather than remove the researcher from the process.
AI generated the evaluation and reasoning, but expert judgment remained necessary to validate the output and determine whether the evaluation accurately represented the user's experience.
Validation & Results
The existing manual process became the benchmark
A useful advantage of this project was that manual PURE scoring already existed.
That gave me an expert-generated benchmark against which the AI-assisted workflow could be evaluated.
I used the existing evaluation process to:
• Compare AI-assisted outputs with expert consensus
• Identify mismatches between AI and human scoring
• Refine the workflow iteratively
• Improve alignment with human judgment over time
Rather than asking whether AI could independently evaluate usability, the more useful question became:
How closely could the AI-assisted workflow support the reasoning designers were already applying?
This kept the system grounded in an established evaluation process rather than treating AI output as inherently correct.
From an hour of evaluation to seconds
The redesigned workflow delivered a significant reduction in evaluation time.
~1 hour
Manual PURE evaluation
↓
5–10 seconds
AI-assisted evaluation
The workflow also:
• Reduced dependency on multi-designer alignment sessions
• Improved consistency across evaluations
• Allowed designers to spend more time on insights and recommendations rather than scoring mechanics
The biggest impact was not simply speed. By reducing repetitive evaluation work, the workflow shifted designer effort toward the parts of research that require human interpretation and judgment.
Learning
The most important lesson wasn't about AI. It was about validating assumptions.
Initially, both my manager and I assumed the system was directly analysing usability-test videos because it appeared capable of doing so.
I challenged this assumption by asking the system to identify precise timestamps corresponding to moments of user difficulty.
It could not reliably extract those timestamps from the video files.
This revealed an important limitation:
The workflow's analysis was primarily driven by text transcripts and structured inputs rather than reliable visual analysis of the recordings.
The video therefore functioned more as contextual input than as a consistently analyzable signal.
This changed how I thought about AI capabilities.
A tool appearing capable of performing a task is not the same as that capability being reliable enough for a real workflow.
For AI-assisted product design, capabilities need to be tested in the context where they will actually be used—not simply accepted based on what the tool appears to support.
Future directions
Where the workflow could go next
The workflow was designed to be modular, allowing additional signals and capabilities to be introduced as the underlying technology becomes more reliable.
Potential extensions include:
01 — Richer usability signals
Incorporating metrics such as error frequency, task abandonment, interaction backtracking, and behavioural indicators derived from user language and hesitation.
02 — Transcript-to-video mapping
Connecting transcript-identified friction points to precise moments in a usability recording as multimodal video analysis becomes more reliable.
03 — Visual evidence extraction
Automatically surfacing screenshots or relevant screen states associated with high-difficulty tasks to make usability issues easier to review.
All future extensions would continue to operate within a human-in-the-loop model, supporting rather than replacing expert judgment.
AI should reduce cognitive load, not replace expertise.
This project reinforced four principles that now shape how I think about AI-assisted product design:
01 — Automate cognitive load, not expertise.
AI is most useful when it removes repetitive evaluation work and gives experts more room for interpretation.
02 — Structure matters.
Reliable AI assistance depends heavily on clearly defined inputs, criteria, and evaluation frameworks.
03 — Validate capabilities.
AI tools can appear capable of performing a task without reliably doing so in the context that matters.
04 — Workflows are products too.
Sometimes the highest-leverage UX problem is not the interface a user interacts with, but the process behind it.
