Stop Picking Metrics Before Reading Failures

Should you begin an eval project by choosing metrics?

Usually, no.

"Helpfulness", "relevance", and "coherence" look sensible in a dashboard. But they may have little to do with the problems your users actually face.

A support bot can sound helpful while quoting an old policy. A research assistant can be coherent while inventing a source. An agent can write a polished reply after updating the wrong account.

Start with the failures, not the metric menu.

Read real traces

Pull 50 to 100 recent conversations or agent runs.

Read them carefully.

For each trace, write one short note describing what went wrong:

• Quoted the old refund policy.
• Asked for information already provided.
• Failed to hand off to a person.
• Used the right tool with the wrong customer ID.
• Gave a correct answer that was too long for mobile.

Do not create categories yet. Just describe the failure in plain language.

This step is sometimes called open coding. The name is less important than the habit.

Group similar failures

After reading enough traces, patterns appear.

Your notes might group into:

• Wrong policy detail: 23
• Did not hand off: 14
• Asked twice for the same information: 10
• Too long for mobile: 5
• Robotic tone: 3

Now you have something useful: a ranked list of actual product problems.

This list tells you where to spend time. It also tells you what to evaluate.

Turn each important failure into a check

Do not build one grader for "quality".

Build focused graders for the failure modes that matter.

For example:

• Policy accuracy: Does every policy claim appear in the supplied document?
• Handoff behaviour: Did the bot transfer the conversation when the user requested a person?
• Repetition: Did it ask for information already present in the conversation?
• Length: Is the response under the mobile limit?

Specific checks are easier to trust and easier to debug.

When the score drops, you know what kind of behaviour changed.

Fix, rerun, and read again

Once the first suite exists:

1. Change the prompt, model, retrieval, or tool logic.
2. Run the same cases again.
3. Compare the failure rates.
4. Read the traces that changed.

This last step matters.

A failed case may mean the system is wrong. It may also mean the task is ambiguous, the reference answer is outdated, or the grader is too strict.

The score points you toward the problem. The trace explains it.

Keep production in the loop

Your first dataset will miss things. That is normal.

Every new incident, complaint, or strange conversation is a candidate test case.

Over time, the process becomes a loop:

Observe → Group → Test → Fix → Ship → Observe again

This is why evals are never really finished. A mature suite is a record of what the product has learned about its own failures.

Three things to remember

1. Generic metrics rarely reveal product-specific failure modes.
2. Read enough traces for patterns to repeat.
3. Every confirmed production failure should become a future test.

The most valuable eval work often happens before anyone writes a grader.

It happens when someone sits down and reads what the system actually did.