What you need to know
An evaluation framework has two parts: what you test (the dataset) and how you score it (the graders). Getting either wrong produces numbers that look reassuring and mean little.
The golden dataset
A golden dataset is a set of realistic inputs with expected outputs or grading criteria, used to judge every version of the system on the same footing. A useful one covers:
| Category | Example for a claims assistant |
|---|---|
| Typical cases | A routine motor claim with all documents |
| Edge cases | A claim with two vehicles and three policy holders |
| Ambiguous inputs | A description that fits two claim types |
| Invalid or incomplete inputs | No policy number; unreadable scan |
| Adversarial inputs | Text that tries to change the assistant's instructions |
| Boundary conditions | The longest documents the system must handle |
| Negative cases | Requests the system should refuse or escalate |
Real production data is the best source of inputs (anonymized as required), weighted to match real traffic, with extra cases for rare but important categories. Synthetic cases fill gaps, especially for adversarial and boundary conditions.
Keep two kinds of suite: capability tests that track progress on what the system cannot yet do well, and regression tests that protect behaviour that already works and must stay working.
The grading ladder
Choose the cheapest grader that is reliable for each behaviour.
| Grader | Use for | Strengths | Limits |
|---|---|---|---|
| Code | Formats, schemas, exact values, banned actions, latency, cost | Fast, cheap, deterministic | Cannot judge meaning or tone |
| Model judge | Faithfulness, completeness, tone, reasoning quality | Scales; handles nuance | Costs money; can be inconsistent; must itself be validated |
| Human review | High-stakes, novel or disputed judgments; calibrating the judge | The reference standard | Slow, expensive, hard to scale |
One system usually uses all three: code checks on every case, a model judge on the interpretive criteria, and humans on a sample and on anything the judge flags as uncertain.
Making a model judge trustworthy
- Give it a detailed rubric with clear definitions and examples of each grade.
- Ask for constrained verdicts (a small set of labels such as pass, minor issue, fail) rather than open-ended scores.
- Ask it to give its reasoning before the verdict, and grade one criterion at a time.
- Calibrate against a set of human-graded cases and track agreement over time.
- Prefer a different model or configuration from the one being graded, to reduce bias toward its own style.
Non-determinism
Model outputs vary between runs. Run important cases more than once, and look at pass rates across runs rather than a single result, especially for agents whose paths can differ.
In the exam
Framework questions ask what the dataset should contain, which grader fits a requirement, or how to make a judge reliable. Strong answers cover hard and adversarial cases, keep the dataset growing from production, apply code first, use a calibrated rubric-based judge for interpretation, and reserve human review for high stakes and calibration.
Check your understanding
A team checks whether every response contains the three required sections. Which grader fits?
Show answer
Answer: B. Presence of fixed sections is deterministic, so code is cheapest and most reliable.
A model judge's scores swing between runs on the same responses. Which change helps most?
Show answer
Answer: A. Detailed rubrics and constrained verdicts make judges more consistent. Wider scales, self-grading and bundling criteria all make variation worse.
Which TWO practices keep a golden dataset useful over time?
Show answer
Answer: A, B. A dataset that learns from production and keeps hard categories represented continues to catch real regressions.
Apply it: Build St. Aldric's evaluation framework
St. Aldric's discharge-summary drafter (lessons 1.5 and 2.4) needs an evaluation framework before the pilot. Design the dataset and the graders.
- Design the golden dataset: size, sources, and the categories it must include.
Done when: the long-stay and high-risk cases have their own slices. - Assign a grader to each requirement: medications match the pharmacy list, pending results are listed, the hospital course is faithful to the notes, no medication changes are suggested, the summary has all required sections, and it is safe to send.
- Make the judge trustworthy: rubric, verdicts and calibration.
- Set the release rule for the pilot.
Show model answer
1. Dataset. 300 past stays, de-identified for the test environment under the privacy officer's rules, sampled to match the real mix of specialties, plus extra cases: 40 long stays (over 80,000 tokens of notes), 30 stays with medication changes during admission, 20 with pending results at discharge, 15 transfers and deaths (summaries with special requirements), and 20 adversarial cases (notes containing copied instructions, contradictory notes). Physician-signed summaries for the same stays serve as references.
2. Graders
| Requirement | Grader |
|---|---|
| Medications match the pharmacy list | Code: compare against the structured list |
| Pending results listed | Code: compare against the EHR's pending list |
| Required sections present | Code |
| No medication changes suggested | Code checks for change language, plus the judge |
| Hospital course faithful to the notes | Model judge with rubric |
| Safe to send | Physician review on a weekly sample and on all judge-uncertain cases |
3. Judge. A rubric with definitions and examples for "faithful", "minor omission" and "unsupported statement"; one criterion per call; reasoning before a three-label verdict; calibrated on 100 summaries graded by two physicians, with agreement tracked monthly. The judge runs on a different configuration from the drafter.
4. Release rule. Zero medication mismatches and zero suggested changes across the set; pending results complete in at least 99%; faithfulness at least 97% overall and no slice below 95%; physician reviewers rate at least 95% of sampled drafts safe to sign with minor or no edits.