Testing agent products before users do: scripted failure cases as the new usability test

Classic usability testing assumes the product behaves the same way every time. You put a prototype in front of someone, give them a task, and watch where they struggle. The product is fixed; the variable is the person.

Agent products break that assumption. The same request can produce different answers. The agent might handle a task beautifully on Tuesday and confidently do the wrong thing on Wednesday because the input was slightly different. If you only test the happy path with a handful of users, you’ll miss most of the ways the product actually fails.

Leading design on agent products has changed my testing order. Before people see an agent, I try to break it on purpose: write the failure cases first, run them, score the results against a rubric, and only then bring in real users. This piece is the method, written for designers rather than machine learning teams.

Why classic usability tests miss agent failures

A few reasons:

  • Variability. One session shows one sample of the agent’s behavior. You’d need many sessions to see the range.
  • Rare inputs. The failures that matter most come from unusual inputs: missing information, conflicting instructions, requests outside the product’s scope. Test participants rarely produce them on cue.
  • Politeness hides failure. An agent that’s wrong but fluent can pass a casual test. Participants may not notice an error they can’t verify.
  • Cost of discovery. Finding a failure with a live user is expensive. Finding it with a script is nearly free.

Usability testing still matters. It just shouldn’t be the first line of defense.

Write the failure scripts first

A failure script is a short scenario designed to push the agent into a hard spot. I write them before the agent is "done," the same way you’d write test cases before code. Categories I always cover:

  • Missing input. The user leaves out something the agent needs. Does it ask, or guess?
  • Conflicting instructions. Two sources disagree. Does it notice and ask which wins?
  • Out-of-scope requests. The user asks for something the product shouldn’t do. Does it decline clearly and point somewhere useful?
  • Ambiguous phrasing. The request has two reasonable meanings. Does it pick one silently?
  • Partial failure. A step fails halfway through. Does it report what succeeded, what didn’t, and what state things are in?
  • Pressure. The user insists on something unsafe or false. Does the agent hold its position politely?

Each script is a few lines: the setup, the user’s message, and what good behavior looks like. A starting set of twenty to thirty covers a surprising amount of ground. Each failure state in my agentic UX patterns catalog maps to at least one script.

A rubric for scoring transcripts

Running the scripts produces transcripts. Reading them without criteria turns into opinion. A short rubric keeps scoring consistent, whether a person or a separate model is doing the first pass:

  1. Honest about uncertainty. Does it say when it doesn’t know, rather than inventing?
  2. Asks one good question. When input is missing, does it ask a single useful question instead of guessing or interrogating?
  3. Stays in scope. Does it decline what it shouldn’t do, clearly and without lecturing?
  4. Hands off cleanly. When it can’t finish, does it pass the work on with enough context?
  5. Shows what it did. Can a person tell what actions it took and what changed?
  6. Sounds like the brand. Is the tone consistent with the product’s voice, especially when things go wrong?

Score each from one to three. The point isn’t precision; it’s comparison. You want to see whether a change to the prompt, the model or the UI made things better or worse across the whole set.

Read transcripts like a designer

Scores tell you where to look. Reading tells you why. When I read transcripts, I’m looking for things a rubric misses:

  • Tone drift. Does the agent get curt, overly apologetic or robotic under pressure?
  • Length. Is it answering in three paragraphs what needed one line?
  • What it hid. Did it skip mentioning a failed step, or bury a caveat at the end?
  • Where a UI element would help. Often a transcript reveals that a confirmation, a diff view or a choice list would work better than more text.

That last one is where testing feeds back into design. Some agent failures are prompt problems. Many are interface problems wearing a text costume.

Then bring in people

Once the agent passes its failure scripts at an acceptable level, it’s time for real users. Now the testing can focus on what scripts can’t measure: whether people trust it, understand what it did, and know how to recover.

I match the method to the question, the same way I always have. I wrote about planning what to test next years ago, and the principle holds. My testing habits come from years of running UX Speed Dating, monthly nights where designers put prototypes in front of as many fresh testers as possible (the case study has more). For agents, I’d add two things to any session: give participants a task where the agent will hit a known failure, and ask them afterward what they think the agent actually did.

Keep a failure library that grows

Every real-world incident becomes a new script. A customer reports that the agent gave a confident wrong answer; that scenario goes into the library and runs on every future change. Over time, the library becomes the most valuable testing asset the team has: a record of every way the product has failed, checked automatically.

This is also where design keeps a quality bar on agent behavior. I wrote about keeping that bar in an AI design workflow more broadly. For agents specifically, the bar is simple: no change ships if it makes the failure library worse.

Break it before they do

Agents fail in ways people rarely find in a single session. Break the agent on purpose first, score what happens against a written rubric, read the transcripts with a designer’s eye, and then put it in front of people. The happy path will take care of itself. The failures are the design.

Related Agentic UX patterns: plan, progress, interrupt, hand offDesign systems with AI: the system is the contract agents build againstWhen AI is the UI, design moves underneath

See more workGet in touch