Use the Simplest Grader That Works

Does this eval really need another LLM?

Sometimes it does.

Often, ten lines of code would be faster, cheaper, and easier to trust.

A practical grading order is:

Code first. LLM judge second. Human review third.

Move down the list only when the simpler option cannot check what matters.

Start with code

Code is the best grader when the requirement is objective.

Examples:

• Is the output valid JSON?
• Does it match the required schema?
• Did the agent call the correct tool?
• Did it use the correct customer ID?
• Does the generated SQL execute?
• Is the answer under 200 words?
• Do the unit tests pass?

These checks are fast, cheap, and deterministic.

They also fail clearly. If the schema is invalid, you know why.

Do not ask an LLM whether JSON is valid. Parse it.

Use an LLM judge for judgment calls

Some requirements cannot be captured with an exact rule.

Examples:

• Is every claim supported by the supplied documents?
• Did the response answer the user's actual question?
• Was the handoff appropriate?
• Does the summary preserve the important meaning?

An LLM judge can handle this kind of open-ended evaluation at scale.

But the prompt must be precise.

"Rate this response for quality from 1 to 5" leaves too much room for interpretation.

A better judge checks one failure mode:

Fail if the reply states a refund window, fee, or eligibility rule that is missing from or contradicts the supplied policy. Otherwise pass.

The second version is easier to explain, test, and improve.

Prefer pass or fail over 1 to 5

Rating scales look more informative than binary decisions.

In practice, people rarely agree on what separates a 3 from a 4. The same reviewer may score the same answer differently tomorrow.

Pass or fail forces the team to define the acceptable boundary.

If one binary score is too simple, split the requirement into several focused checks:

• Factually supported: pass or fail
• Correct handoff: pass or fail
• Required details present: pass or fail
• Length limit respected: pass or fail

You get more useful information without pretending that a vague number is precise.

Use humans where they add the most value

Human review is still essential.

Use domain experts to:

• define what good looks like
• label examples for judge development
• resolve ambiguous cases
• spot-check automated results
• review high-stakes decisions

Humans should not grade every production output. That does not scale.

Their role is to create and protect the standard that automated graders apply.

One trusted domain owner is often more effective than a large committee. Someone needs to make the final call when reasonable people disagree.

Combine graders when needed

A single case can have several graders.

For a support response, you might check:

1. Valid output structure with code
2. Policy grounding with an LLM judge
3. A random sample with a human reviewer

Each grader handles the part it is best suited for.

This is usually better than asking one large judge prompt to evaluate correctness, tone, safety, length, and relevance at once.

Three things to remember

1. If code can check it, use code.
2. Give an LLM judge one specific failure mode.
3. Use humans to define and audit the standard.

The best grader is not the most advanced one.

It is the simplest grader that can make the decision reliably.