How do you know if an AI response is good? Evaluating with test sets and LLM-as-judge

Building a prompt or setting up a language model application is often sorted out within a few minutes. The real challenge, however, only begins when you want to determine whether the generated answers are actually correct, complete, and safe. This article falls within Module 2 of the learning track, focused on steering, using, and qualitatively evaluating AI models in practice. Where traditional software behaves deterministically — the same input produces exactly the same output — large language models exhibit probabilistic behavior. This makes quality assurance a complex challenge.

Judging answers by eye carries significant risks. A change to a prompt can improve the answers to ten specific questions while unnoticeably degrading quality on fifty other questions. To bring applications to production in a scalable and reliable way, a structured evaluation methodology is indispensable. In this guide, we cover how to build a representative test set, which traditional metrics are useful, and how you LLM-as-a-judge to automatically quantify the quality of complex AI responses.

The problem of subjectivity and scale in manual evaluation

When a team starts developing an LLM application, evaluation is almost always done manually at first. Developers enter a number of prompts, read the result, and judge intuitively whether the outcome is good enough. This method, also known as "vibe checking", is extremely useful in the very first exploration phase. It helps quickly spot gross errors and strange behavior. However, once the application is deployed to hundreds of users for a wide range of tasks, this approach falls short.

Manual evaluation has three fundamental bottlenecks:

• Lack of scalability: A human reviewer can thoroughly check at most a few dozen answers per hour. With continuous updates to prompts, system instructions, or retrieved documents, it's impossible to manually re-run hundreds of test cases.
• Inconsistency and subjectivity: Two different reviewers rarely interpret the quality standards identically. Even the same reviewer may judge more strictly on Monday morning than on Friday afternoon. This makes comparing model versions unreliable.
• Blindness to regression: Without a fixed set of hundreds of categorized test questions, you won't notice that an apparent improvement in style leads to a deterioration in factual accuracy on rare edge cases.

A well-thought-out test framework replaces gut feeling with measurable data. It enables teams to transform measuring and improving into a continuous, iterative engineering cycle.

The foundation: building a representative test set

No evaluation method is worthwhile if the underlying test set doesn't reflect reality. A test set, often called a golden dataset or evaluation benchmark, is a collection of structured test cases used to measure the performance of the AI system. A good test set doesn't consist of random questions, but is carefully assembled to cover the full spectrum of expected user interactions.

An effective test case contains at least the following elements:

1. Input: The exact prompt or question presented to the model, optionally supplemented with system instructions or historical conversation context.
2. Context (optional): For RAG systems (Retrieval-Augmented Generation), this includes the source documents provided to the model as an information source.
3. Reference answer (Ground Truth): The ideal, expert-verified answer to the question.
4. Evaluation criteria: Specific guidelines against which the output must be judged, such as factual accuracy, conciseness, tone, or safety.

For a realistic picture, a test set should consist of at least three categories of questions. First, frequent questions: the typical tasks that make up 80% of daily interactions. Second, complex edge cases: questions with ambiguity, double negatives, or requests that push the limits of the context. Third, adversarial questions: deliberately misleading or provocative prompts that test whether the system refuses politely or maintains the right boundaries.

For setting up structured tests at production level, it's wise to consult the article on how you systematically test prompts before they go live. It goes deeper into integrating these datasets into automated test pipelines.

Deterministic and heuristic metrics: pros and cons

Before the rise of extremely powerful language models, the field relied mainly on deterministic and statistical text-based metrics to compare generated text with a reference answer. These methods compare the literal overlap of words or syllables and are extremely fast and cheap to run.

Although metrics such as ROUGE and BLEU remain useful for specific tasks such as comparing JSON output or checking strict syntax, they fall short when judging natural language. An AI answer can be 100% factually correct in substance, yet show zero overlap with the reference answer due to a different sentence structure or synonym use. Conversely, an answer can use exactly the same words as the original, but completely reverse the meaning through a misplaced comma or negation. More flexible methods are needed for semantic evaluation.

Model-based evaluation: the LLM-as-a-judge pattern

To work around the limitations of statistical metrics without falling back on slow human review, the industry has embraced the LLM-as-a-judge design pattern. Here, an advanced and powerful language model (such as GPT-4o or Claude 3.5 Sonnet) is deployed as an automatic judge of answers generated by another (often smaller or cheaper) model.

The evaluator LLM is given a structured prompt containing three elements: the original question, the generated answer to be judged, and an explicit evaluation framework (rubric). The judging model analyzes the answer and returns a score, accompanied by a substantive justification.

[INSTRUCTIE VOOR DE EVALUATOR]
Je bent een onpartijdige, deskundige beoordelaar. Beoordeel het onderstaande antwoord op feitelijke juistheid en juistheid ten opzichte van de meegegeven bron.

BRONTEXT:
"De klant heeft recht op 14 dagen bedenktijd, mits het product onbeschadigd is."

GEGENEREERD ANTWOORD:
"U kunt het artikel binnen twee weken retourneren als het nog in nieuwstaat verkeert."

CRITERIA:
- Score 1: Antwoord spreekt de bron tegen of bevat hallucinaties.
- Score 2: Antwoord is gedeeltelijk juist maar mist cruciale voorwaarden.
- Score 3: Antwoord is feitelijk juist en sluit volledig aan op de bron.

Geef je beoordeling in JSON-formaat met de sleutels 'redenering' en 'score'.

With this pattern, hundreds of answers per minute can be substantively checked on aspects such as relevance, organizational tone, correctness of the steps, and the presence of hallucinations. The power lies in the evaluation model's ability to correctly assess rephrasing and synonyms on their merits.

Prompting techniques and evaluation frameworks for the judge

The success of LLM-as-a-judge stands or falls with the quality of the judging prompt. If the instructions given to the evaluation model are vague, the scores become just as unreliable as human grades. To achieve a high degree of consistency, specific prompting techniques are applied.

The most important pillars for a robust evaluator prompt are:

1. Provide clear rubrics and scales

Avoid open-ended questions such as "Is this answer good?". Instead, use explicitly defined rubrics. A scale of 1 to 3 or 1 to 5 works better in practice than a scale of 1 to 100. On a scale of 1 to 100, the difference between a 72 and a 78 is completely arbitrary. On a scale of 1 to 3, every score has a clear meaning (for example: 1 = Wrong/Dangerous, 2 = Incomplete, 3 = Correct and complete).

2. Force reasoning before the score (Chain-of-Thought)

Just as with normal prompts, the accuracy of the assessment improves enormously when the evaluator is required to first formulate the arguments step by step before giving the final grade. If the model picks a grade first, it will tend to justify that grade for the rest of the text, even if the analysis shows the grade was wrong.

3. Evaluate specific dimensions separately

Don't try to capture all quality aspects in a single score. Have the model perform separate assessments on different dimensions:

• Faithfulness: Is the answer based solely on the provided source context, without adding fabricated facts?
• Answer Relevance: Does the answer actually address the question that was asked, or does it wander off topic?
• Context Precision: Does the retrieved source context contain only relevant information, or is there a lot of noise mixed in?
• Tone and Safety: Does the answer comply with the correct style guidelines and does it contain no harmful content?

Pitfalls of LLM-as-a-judge: recognizing systematic bias

Although LLM-as-a-judge is extremely powerful, the judging model is itself also a language model with built-in quirks and biases. Anyone who doesn't recognize these pitfalls runs the risk of making decisions based on skewed or distorted metrics.

The four most common forms of bias in evaluator models are:

1. Positional Bias: When you ask an LLM to compare two answers (A vs. B), the model has a strong tendency to prefer the answer presented first (Option A) or, alternatively, the one presented last. This phenomenon can be countered by running every comparison twice with the order reversed and only accepting results that remain consistent.

2. Verbosity Bias: Large language models have a strong preference for long, elaborately formulated answers. A concise, perfectly correct answer often gets a lower score from a judge than a rambling, long-winded answer that looks professional. This can be corrected by explicitly stating in the rubric that unnecessary padding leads to a point deduction.

3. Self-Enhancement Bias: A model developed by a particular vendor tends to rate answers generated by the same model (or within the same model family) higher than answers from competing models. It's therefore wise to use a different, neutral model as the judge.

4. Egocentric/Leniency Bias: Without strict instructions, evaluation models tend to give grades on the high side (for example, mostly 4s and 5s on a 5-point scale). By including strict negative examples (few-shot examples) in the prompt, you force the evaluator into a more critical stance.

For a good overview of specialized software, frameworks, and open-source packages that automatically correct this bias and simplify evaluation loops, you can check the overview of evaluation and testing tools for LLM applications (evals).

Hybrid evaluation strategies and continuous monitoring in production

No single evaluation method is sufficient on its own. The most successful AI teams use a hybrid strategy in which different forms of evaluation are combined in a pyramid model.

At the base of the pyramid are statistical unit tests (such as JSON validation and keyword checks). These run within seconds on every code change. The middle layer consists of automated LLM-as-a-judge tests on a test set of several hundred representative questions. These run nightly or before a new release. At the top of the pyramid is human spot-check review and monitoring of direct user feedback (such as thumbs up/down buttons in the application).

Besides pre-deployment testing, continuous monitoring (observability) in production is essential. By running a lightweight LLM-as-a-judge evaluation on a sample basis in production (for example, 5% of all incoming conversations), sudden deviations in quality or the emergence of new types of questions are spotted immediately.

Setting up such evaluations, however, comes with infrastructure and token costs. If you want to keep the financial impact of large-scale evaluations under control, read the in-depth article on the sister domain about how you can control the costs of evaluation in LLM applications.

Summary and practical checklist

Evaluating AI responses requires a shift from intuition to systematic measurement. By building a representative test set, combining traditional metrics with scalable LLM-as-a-judge evaluations, and keeping an eye on built-in bias, you create a reliable quality foundation for any AI application.

Use these steps as a starting point for your evaluation pipeline:

• Assemble a fixed test set with at least 50 to 100 representative real-world questions and approved reference answers.
• Use deterministic metrics for hard format requirements (such as JSON or specific code structure).
• Deploy a powerful language model as an evaluator for substantive accuracy, relevance, and tone.
• Define explicit rubrics and have the judge reason first before assigning a score.
• Swap the order of answers in comparative evaluations to neutralize position bias.
• Perform spot-check human reviews to validate the reliability of your judge prompt.