What belongs in an eval dataset?
Include common tasks, edge cases, unsafe requests, historical failures, and the context needed to score each result. Every row should state what success means, whether through an exact answer, a rubric, a tool outcome, or a human label. An AI evaluation harness runs these cases consistently, and an LLM as a judge can score the cases that have no single right answer.
How should an eval dataset be maintained?
Version cases and labels, review disagreements, and keep test items out of training or prompt examples. Add confirmed production failures without letting recent incidents dominate the whole set. Synthetic data can fill sparse scenarios, but validate it against real behavior. Use online and offline evaluation together, and monitor benchmark contamination whenever evaluation examples can leak into model development.
How do we build eval datasets for client systems?
We pin the expected outcome of every case to a person’s sign-off. A regression baseline is the human-signed expected outcome, never whatever passed last run, because re-baselining on the last run’s output lets slow drift hide inside the baseline. We also rerun the golden set when the rules, registries or source data change, not only when code ships: retrieval and conflict handling can break silently when the data underneath moves.
For the test that a human correction resurfaces on a near-identical input, we climb three steps. First, a single integration test. Second, when we tune retrieval settings, a small experiment over inline cases with a deterministic recall check. Third, a managed dataset, only once subject matter experts are curating scenarios. One trap we design around: an eval task that writes to a shared store lets one case contaminate the next, so each case needs isolated data or a clean store.
What should an eval dataset not test?
Keep deterministic checks out of it. Retrieval correctness, permission scoping and citation validity can be asserted in ordinary tests that run on every change. Spend the labeled dataset on what needs judgment: answer quality, faithfulness to the sources, and whether the system knew when to escalate, scored by people or by a calibrated judge as in our post on LLM as a judge in LangSmith.