What Happens When the Model Under Your App Changes?
The model behind your application changes.
Maybe a new version is available. Maybe a cheaper model looks good. Maybe the provider updates a model snapshot behind the same name.
How do you know your product still works?
You rerun the same eval suite.
This is where evals stop being a one-time experiment and become part of the product lifecycle.
While prototyping: read outputs
Do not build heavy infrastructure for a weekend prototype.
Run the product. Read 20 outputs. Keep a simple list of what breaks.
This is already a form of evaluation.
The purpose at this stage is to learn quickly, not to create a dashboard.
Before launch: build the first real suite
Read a larger set of traces and group the common failure modes.
Build 20 to 50 cases around the most important ones. Add structured synthetic cases where production data is missing.
Set a clear release bar.
For example:
- no critical policy violations
- at least 90% on the capability suite
- 100% on critical regression cases
- latency and cost within limits
The exact numbers depend on the product. The important part is deciding before launch, not after seeing the score.
On every change: protect solved behaviour
Once a capability works reliably, move its cases into a regression suite.
Run that suite when you change:
- the prompt
- retrieval logic
- tool definitions
- application code
- model or provider
Put the suite in CI where possible. A pull request that breaks solved behaviour should show which cases flipped before it merges.
This is the AI equivalent of keeping a bug from returning.
When the model changes: compare side by side
Model upgrades should not rely on a general leaderboard or a few demo prompts.
Run the old and new models against the same cases.
Compare:
- failure rate by category
- newly fixed cases
- newly broken cases
- latency
- token use and cost
- judge consistency
A cheaper model may be the better product choice if quality stays within the required bar.
A more capable model may be a bad upgrade if it introduces regressions in a critical workflow.
In production: evaluate a sample continuously
Offline datasets never capture everything users will do.
Sample real production traces and run focused graders on them. Track failure rates by category, not only one overall quality score.
Send unusual or high-risk cases to a human review queue.
Watch trends. A slow increase in one failure mode can reveal a data change, a prompt issue, or a provider change before it becomes a major incident.
After every incident: add a test
An incident should improve two things:
- The product
- The suite that protects the product
The bad input becomes a regression case. The expected behaviour becomes explicit. The fix is not complete until the case passes and stays in the suite.
Over time, the suite becomes institutional memory.
It records the failures your team has already paid to understand.
The tool is not the practice
You can begin with a spreadsheet and a script.
Later, you may add a library for CI, a tracing platform, an annotation queue, and online evaluation.
Those tools help with scale. They do not decide what good means.
Teams with a strong dataset and simple tooling usually learn more than teams with a sophisticated platform and vague metrics.
Three things to remember
- The evaluation process should grow with the product.
- Every model or prompt change should run against the same important cases.
- Every production incident should strengthen the regression suite.
Models will keep changing.
Your definition of acceptable product behaviour needs to remain stable enough to compare them.