Skip to content
Home › Prompt Engineering & Structured Output

Building Evals

Prompt Engineering & Structured Output

Definition

An evaluation ("eval") is a repeatable test set that scores prompt/model output against expected answers, letting you measure prompt quality and iterate deliberately instead of eyeballing results.

Key points

  • Three grading strategies:
    1. Code-based / exact match — deterministic checks like output == golden_answer (or classification comparisons); cheapest and most reliable when the answer is exact.
    2. Model-based (Claude-as-judge) — Claude grades Claude's output against a rubric; use for open-ended answers where exact match is too brittle.
    3. Human grading — manual review (e.g. via Anthropic's Workbench) for subjective/open-ended tasks that resist automation.
  • Claude-as-judge grader prompt: have the judge reason in <thinking></thinking>, then emit a verdict in <correctness></correctness> as correct / incorrect. Rubric rule: "An answer is correct if it entirely meets the rubric criteria, and is otherwise incorrect." Extract with a regex like r"<correctness>(.*?)</correctness>".
  • Eval set structure: a list of dicts, each with an input field (e.g. question, animal_statement) and a golden_answer — an exact value for code grading, or rubric text for the judge.
  • Score idiom: accuracy = grades.count('correct') / len(grades) * 100%.
  • Tuning: constrained verdicts use a tiny max_tokens (e.g. 5); open-ended generations use a larger budget (e.g. 2048).
  • Eval-driven iteration: change the prompt, re-run the eval set, compare scores — evals turn prompt engineering into a measurable loop rather than guesswork. Frameworks like Promptfoo support code-graded, classification, and model-graded evals with custom graders.

Why it matters for the exam

  • Distinguishing code-graded vs model-graded vs human grading (and when each applies) is directly tested in the Prompt Engineering domain.
  • The Claude-as-judge rubric pattern (<thinking> reasoning + <correctness> verdict) and the accuracy formula are concrete, testable idioms.

Common gotchas

  • Reaching for a model-based grader when a cheap exact-match code check would do (extra cost + non-determinism).
  • Forgetting that a Claude-as-judge verdict must be extracted from a fixed tag — free-form judge output is not machine-scoreable.
  • An answer is judged correct only if it entirely meets the rubric; partial matches are incorrect.

See also

Sources

Referenced by