Most teams have a careful process for shipping code. Pull requests, reviews, tests, staging. Then someone changes one line in a config file from one model to another, and the whole product behaves differently in production by lunchtime.
Swapping models is usually done for good reasons. A newer version is cheaper, faster or scores better on public leaderboards. But public leaderboards measure general ability, not your product. A model that is better on average can still be worse at the three things your customers actually ask about.
What actually changes
In the swaps we’ve assayed, the regressions rarely show up as obvious nonsense. They show up as small shifts in behaviour that add up:
The new model follows formatting instructions more loosely, so downstream parsing breaks on a few percent of answers.
It is more eager to help, which means it fills gaps in your documentation with confident guesses.
It refuses different things. Some safe requests start getting declined, and some unsafe ones stop being declined.
Token usage changes, so the cheaper model is not always cheaper per conversation.
A model that is better on average can still be worse at the three things your customers ask about most.
How to test a swap
The fix is not complicated. Run the same suite against both models, side by side, before the switch. Compare scores per category, not just in total, and read the cases where the two models disagree. Those disagreements are where the real behaviour change lives.
If the candidate wins, great: switch with evidence. If it loses in one area, you now know exactly which prompts or guardrails to adjust before it goes live. Either way, the decision stops being a guess.
When we started, our reports showed pass rates as percentages. Customers would read “97% passed” and relax. That was the problem.
At ten thousand scenarios, three percent is three hundred failures. Some of them might be harmless formatting issues. Others might be the assistant reading out another customer’s address. A single percentage flattens those into the same number and makes a dangerous release look almost perfect.
Borrowing from the assayers
Gold has been graded in parts per thousand for centuries. Fineness 999 means almost pure; 750 means a quarter of the metal is something else. The scale is precise enough to show small differences and familiar enough that people read it without a manual.
Our fineness score works the same way, with one important difference: failures are weighted by severity. A leaked record costs far more points than a slightly off tone. The result is a number from 0 to 999 that moves sharply when something serious breaks and barely twitches for cosmetic issues.
A percentage tells you how often you were right. Fineness tells you how much it would cost when you were wrong.
One number, many readers
The score is deliberately simple at the top and detailed underneath. An engineering lead sets a release gate at 900. A product manager sees the trend across releases. A risk officer sees that every release since March was assayed and passed. And anyone who wants the detail can click into the exact failing cases behind every lost point.
We still show the raw pass rates, by category, inside every report. But the first thing anyone sees is the fineness, because that’s the number that should decide whether you ship.
When people think about attacks on AI assistants, they picture a user typing a clever jailbreak into the chat box. That happens. But in the products we test, the more dangerous path is quieter: instructions that arrive inside content the assistant reads on its own.
Retrieval systems pull in help-centre articles, PDFs, emails, tickets and web pages. Agents read tool outputs and API responses. Any of that text can contain instructions, and the model has no reliable built-in way to tell the difference between “data I’m reading” and “orders I should follow”.
What it looks like in practice
A customer uploads an invoice with a line of white-on-white text telling the assistant to approve a refund.
A public web page your agent browses asks it to include a link in every answer.
An old internal document in the knowledge base contains a draft policy that overrides the real one.
If your assistant reads it, someone can write to it.
Three tests to run this week
You don’t need a platform to start. Take your three most common document types and plant a harmless canary instruction in each, such as asking the assistant to end every answer with a specific word. Then ask ordinary questions that cause those documents to be retrieved.
If the canary word shows up in answers, your product follows instructions from content. Next, test whether it will reveal its system prompt when a document asks it to. Finally, if you run an agent, check whether a document can make it call a tool the user never asked for.
These three checks won’t cover everything, but they will tell you very quickly whether you have a problem. In our experience, most teams do, and most are surprised by how little it takes.
RaunstenLABS