LangSmith’s evaluation concepts describe offline datasets and online production traces as complementary parts of one improvement loop.
When should you use offline evaluation?
Before every release. Use an AI evaluation harness to compare prompts, models, retrieval settings, or workflow logic against stable examples. Offline tests provide repeatability and reference outputs, which makes them suitable for regression gates and for calibrating LLM judges against human labels. Their limit is that they cover only what someone thought to test.
What does online evaluation add?
Production data exposes new inputs, changing behavior, and failures the test set missed. LLM observability supplies the runs, and AI agent evals can inspect the full agent trajectory. Online judges usually score a sample to control cost: LangSmith’s guide gives a sampling rate of 0.1 to evaluate 10% of traces (LangSmith online evaluations, read 2 November 2026). Online evaluation is still the rarer practice: in LangChain’s survey of 1,340 respondents, 52.4% run offline evaluations and 37.3% run online ones (State of Agent Engineering, read 2 November 2026).
How do we combine online and offline evaluation at Soba Labs?
For the AI sales agent we built for Open Loyalty, a small evaluation set drawn from representative real conversations gated every change to the prompt, model, retrieval settings, or tool schema. It checked routing, field extraction, grounding in approved material, and unsafe or duplicate side effects. Production failures were classified and the best examples promoted into the set, so a duplicate CRM write found in a trace became a permanent simulated case (LangGraph agents in production).
The online half earned its place on a retrieval pipeline we shipped. It passed every offline check and looked clean for weeks, until someone flagged an answer citing a superseded policy; the traces showed the pattern from the first week, at a rate too low for spot checks. That is why we wire calibrated online judges, covered in LLM-as-judge evaluation in LangSmith, and rerun the golden set when the data changes, not only when the code deploys. Offline tells us a change is safe to ship; online tells us the world has moved.
Which should a team build first?
Offline, because a gate on every change is cheap and catches the regressions you already know about. Add online scoring as soon as real traffic arrives, starting with the failure you can least afford. The useful loop sends online failures back into the offline dataset, verifies a fix there, and then watches production for recurrence. Neither half is enough alone: an agent with traces but no eval gate can watch itself regress and still ship the regression.