A Clever Grader Cannot Save a Weak Dataset

Would your eval catch a question nobody on your team thought to write?

Probably not.

That is why the dataset matters so much.

A sophisticated grader applied to easy, repetitive questions gives you a precise answer to the wrong problem.

The quality of an eval begins with the cases you choose.

Start with real behavior

The best test cases usually come from places where the product has already met reality:

1. Production conversations
2. Bug reports and complaints
3. Domain experts
4. Synthetic data

Production data is valuable because users are strange in ways a planning meeting cannot predict.

They make typos. They combine three requests. They leave out key details. They ask the same question in five different ways.

Bug reports are even better. Each one gives you a real input and a known failure. Once fixed, it should become a regression case.

Add synthetic cases with structure

Before launch, you may not have enough production data.

Synthetic data can help, but a broad prompt such as "Generate 50 customer questions" usually produces polite, similar, easy examples.

Instead, define the dimensions that make cases meaningfully different.

For a refund assistant, those dimensions might be:

• Customer type: new, returning, business
• Intent: refund, exchange, invoice
• Situation: within policy, outside policy, missing receipt
• Tone: neutral, angry, confused
• Channel: short mobile message, email, chat transcript

Then combine the dimensions before generating the final wording.

This produces better coverage because the variety comes from the scenario, not from superficial phrasing.

Include cases the system should not answer

Many datasets contain only answerable questions.

That rewards a dangerous behavior: answering everything confidently.

Include cases where the correct action is to:

• refuse
• say "I do not know"
• ask a clarifying question
• hand off to a person
• avoid calling a tool

The absence of an action can be the correct outcome.

Your dataset should measure restraint as well as capability.

Write down what good means

Every case needs a target.

That target does not always need to be a perfect reference answer. It can be:

• facts that must be mentioned
• claims that must not appear
• an expected tool call
• an expected database state
• a required refusal or handoff

Clear expectations make grading easier and expose disagreements early.

If two experts cannot agree on what should happen, the problem is not ready for automated grading.

Keep a hidden set

If developers can see every eval case, they may tune the prompt until it passes those exact examples.

The score improves, but general behavior may not.

Keep part of the dataset hidden. Use it as an honest check after tuning.

Also version the dataset. A score from version 4 should not be compared directly with a score from version 7 if the cases changed substantially.

Start small, then earn complexity

Twenty to fifty strong cases are enough for a useful first suite.

The goal is not to collect thousands of questions immediately. The goal is to create a set that reflects real risk and teaches you something when it fails.

Grow it with every incident, every new feature, and every new class of user behavior.

Three things to remember

1. Real user traces are usually your best source of cases.
2. Synthetic data needs structured dimensions, not one broad prompt.
3. A strong dataset includes cases where the correct behavior is restraint.

The grader tells you whether the system passed the test.

The dataset decides whether you asked the right question.