What is ground truth in machine learning?
In supervised learning, ground truth is the set of labels attached to examples: spam or not, the invoice total, the defect region in an image. The model trains on one portion and is scored on a held-out portion, as model validation requires. The term is older than machine learning: remote sensing used it for measurements taken on the ground to calibrate aerial and satellite data. The machine learning sense is looser: the labels count as ground truth even when they are subjective or inaccurate (Wikipedia, “Ground truth”, read 2 November 2026). Ground truth is a decision about what counts as correct, not a fact the data hands you. Producing it is the work of data annotation.
How does ground truth work in AI and agent evaluation?
Each row of an eval dataset pairs an input with its ground truth, and the form depends on the task. A classifier has one correct label. An extraction task has expected field values. An agent has a verifiable end state: the right record updated, the right API called. Open-ended text has no single correct answer, so the ground truth becomes a reference answer plus a rubric, graded by people or by an LLM as a judge that was itself checked against human labels. Use exact checks wherever a verifiable answer exists, and spend judgment only where it does not. This split is the backbone of AI agent evals.
How do you build reliable ground truth?
Write labeling guidelines with worked examples of hard cases. Have two people label the same sample independently and measure agreement; where they disagree, an expert adjudicates and the guideline gets clearer. Version the labels like code, and keep test labels out of training and prompt examples to avoid benchmark contamination. Model scores above the annotators’ own agreement rate cannot be measured reliably, because past that point the metric tracks one annotator’s taste.
What happens when ground truth arrives late?
In production the true outcome often trails the prediction. A fraud label comes with the chargeback weeks later, a loan default months later, a churn label at renewal. Until it arrives, teams monitor proxies such as input distributions, confidence scores and human review samples, then backfill real outcomes into the metrics as they land, which is how model drift is confirmed rather than guessed. Plan the label pipeline before launch, or live accuracy stays unknowable for exactly as long as the outcome takes to happen.