What Are AI Evals? How to Measure LLM App Quality in 2026

Guardrails Block Bad Output. Evals Tell You Whether Anything Improved.

If you have read our guide to AI guardrails, you know the runtime layer: input filters, output validators, schema checks, PII redaction, refusal handling. Guardrails are a control system. They sit in the request path, fire on a single response, and their verdict is binary - let this through, or do not.

Evals are a measurement system. They sit outside the request path, run across a whole set of cases at once, and produce a number you can compare between versions.

The consequence teams learn late: a guardrail cannot detect a regression. It has no baseline and no memory. If your new model version is subtly worse at grounding answers in retrieved documents but still returns valid JSON and nothing toxic, every guardrail you own waves it through. Guardrails protect the user in the moment. Evals protect the product over months.

What an Eval Actually Is

Strip away the dashboards and an eval has exactly three parts.

A dataset of test cases. Rows with an input, optionally a reference output (the ground truth), and optionally metadata such as tags, difficulty or source trace ID. Every serious platform models this identically: LangSmith calls them datasets and examples, Braintrust calls it data with inputs and optional expected outputs, Inspect calls them datasets of labelled samples.

A task. The thing under test. Braintrust describes it as the function being evaluated - typically an LLM call, but equally a multi-step agent or a retrieval pipeline. Inspect calls this a solver.

A scorer. A function turning input, output and reference into a number or label: exact match, embedding similarity, a regex assertion, a rubric applied by a judge model, or a human rating. Inspect calls them scorers, Braintrust scorers and classifiers, LangSmith evaluators. Same primitive.

Run the task over the dataset, apply the scorer to every row, aggregate. That is an eval. Everything else - traces, CI hooks, annotation queues, drift alerts - is convenience built around those three pieces.

One property separates a real practice from a one-off spreadsheet: repeatability. You must be able to run the identical set against a new prompt, model or retriever and get a number you can honestly compare.

The Main Types of Eval

Most mature setups combine three: assertions as a fast gate, a judge for open-ended quality, and a human-labelled slice to keep the judge honest.

Which Metrics Matter for Your Task Type

Generic quality scores are close to useless. Pick metrics that match what your system actually does.

Retrieval and RAG

If you are building on retrieval-augmented generation, the failure is rarely the model alone - it is retrieval feeding it the wrong context. Ragas publishes the vocabulary most of the ecosystem borrowed: context precision (is retrieved context relevant), context recall (did you retrieve everything needed), context entities recall, noise sensitivity (how badly irrelevant chunks derail the answer), faithfulness (is the answer supported by the context), and response relevancy.

That split matters diagnostically. Low context recall is a chunking or embedding problem. High context recall with low faithfulness is a generation problem. Our walkthrough on debugging a RAG pipeline that returns irrelevant results covers how to act on each signal.

Agents

Agent evals score the path, not just the destination. Ragas ships tool call accuracy, tool call F1, agent goal accuracy and topic adherence. DeepEval covers similar ground with task completion, tool correctness, goal accuracy, step efficiency and plan adherence. Both separate trajectory from outcome because agents routinely reach the right answer by a wasteful or unsafe route.

Classification, extraction and safety

For classification use the metrics that predate LLMs: accuracy, precision, recall, F1, per-class confusion matrices. Add a structural check, since a schema violation is a different bug from a wrong label - DeepEval includes a JSON correctness metric for this. On safety, DeepEval exposes hallucination, bias and toxicity metrics, while promptfoo attacks the same surface adversarially through red-teaming. If hallucination is your main risk, our guide on detecting and preventing hallucinations in production goes deeper than any single metric can.

LLM-as-a-Judge: Useful, and Quietly Biased

Judges exist because reference answers do not scale. Nobody hand-writes a golden response for every support conversation. A judge model reads the output, scores it against a rubric, and does so cheaply at whatever volume you need.

The evidence that this works is real but bounded. The MT-Bench paper, Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena by Zheng and colleagues, reports that strong judges such as GPT-4 reach over 80 percent agreement with both controlled and crowdsourced human preferences - the same level at which humans agree with each other. G-Eval reports a Spearman correlation of 0.514 with human judgement on summarisation, which its authors note outperformed previous methods by a large margin. Read that honestly: 0.514 is a moderate correlation and a genuine improvement, not a solved problem.

The same literature names the failure modes. MT-Bench documents position bias (the judge favours whichever answer it sees first), verbosity bias (longer answers score higher regardless of quality), and self-enhancement bias (the judge favours its own outputs). Panickssery and colleagues went further in LLM Evaluators Recognize and Favor Their Own Generations, showing that models including GPT-4 and Llama 2 can identify their own outputs at non-trivial accuracy, with a linear correlation between that self-recognition ability and the strength of self-preference.

Calibrating a judge is not complicated, just unglamorous:

• Hand-label a sample first. Around a hundred rows scored by someone who knows the domain. This is ground truth for the judge itself.
• Measure agreement against those labels. If the judge disagrees with your experts more than your experts disagree with each other, the rubric is the problem.
• Swap positions in pairwise comparisons. Run A-then-B and B-then-A, and flag any pair where the verdict flips. That is position bias made visible.
• Control for length. If a 600-word answer outscores a 150-word answer saying the same thing, you have verbosity bias, not quality signal.
• Do not let the generator judge itself. Use a different model family on any decision that gates a deploy.
• Re-calibrate when the judge model changes. A judge upgrade silently rebases every historical score you have.

Building an Eval Set Worth Trusting

Source from real traces. The highest-value rows come from production: the ticket where the agent looped, the question where retrieval returned nothing useful, the answer a customer flagged. Observability platforms exist partly to make this harvesting easy - see our comparison of LangSmith, Langfuse, Braintrust, Helicone and Phoenix for how tracing feeds evals. Synthetic cases are fine for breadth, but a set built only from imagination measures a product you do not have.

Grow it from bugs. Treat every production failure like a normal software bug: reproduce, add to the suite, then fix. Over a year this compounds into a set shaped like your real risk surface.

Avoid leakage. If the cases in your eval also appear in your few-shot examples, prompt library or fine-tuning data, your score measures recall of the answer key. Keep a held-out slice you never look at while iterating.

Version it like code. The eval set belongs in source control or a platform with dataset versioning, with a changelog. A score from March and a score from July are not comparable if the dataset silently grew by forty rows in between.

Where Evals Run

In CI, as a gate. Any change to a prompt, model version, temperature, retriever or chunking strategy triggers the suite. DeepEval runs via a deepeval test run command; LangSmith documents pytest and Vitest or Jest integrations; promptfoo automates checks in CI/CD; Braintrust frames CI/CD regression catching as one stage of its cycle. Most of your value comes from here, because it is the only layer that runs whether or not anyone remembers to.

Pre-deploy, as a heavier check. Judge-based and human-in-the-loop passes too slow for every commit run before a release.

Online, as continuous sampling. Score a percentage of live traffic with reference-free evaluators. LangSmith splits exactly this way: offline evaluators compare against reference outputs, online evaluators assess quality without them. This catches inputs you never anticipated - which is most of them.

The Eval Tooling Landscape in Mid-2026

Housekeeping first, because two different things share a name. The openai/evals GitHub repository is an MIT-licensed framework and benchmark registry, and it remains available. OpenAI's hosted Evals platform - the dashboard and Evals API - is a separate product and is being retired. Per OpenAI's own deprecations page, the deprecation was announced on June 3 2026, existing evals go read-only on October 31 2026, and the dashboard and API shut down on November 30 2026, with a documented migration path to promptfoo. If you have eval definitions, graders or historical runs there, export them before the read-only date.

Also worth knowing

LangSmith publishes the clearest conceptual model of the group - datasets and examples, experiments, evaluators, runs and threads - with LLM-as-judge, pairwise comparison, annotation queues and pytest or Vitest integrations; its pricing page lists a free Developer tier and a Plus tier at 39 dollars per seat per month at the time of writing, self-hosting on Enterprise only. Inspect, the MIT-licensed framework from the UK AI Security Institute, suits teams wanting formal evaluation structure over a product dashboard. Arize Phoenix offers OpenTelemetry-based tracing with response and retrieval evals and is self-hostable, though under the Elastic License 2.0 rather than an OSI-approved licence.

Common Failure Modes

The eval set is too small. With a dozen rows, one flipped case moves the aggregate several points and you spend a week chasing noise.

The judge was never calibrated. Nobody checked it against human labels, so the number is precise and meaningless. This is the most common way an eval program produces confident nonsense.

Vanity metrics. An average helpfulness score that has never triggered an action is a dashboard, not an eval. Every metric needs a threshold and a decision attached.

Overfitting to the eval. Weeks of tuning against a frozen dataset produce a rising score and flat user satisfaction. Rotate in fresh traces and hold out a slice.

No regression suite. Teams run evals when they suspect a problem, so they never catch the problems they do not suspect. If it is not wired into CI, it will not run.

Scoring only the final answer. For agents especially: a correct answer reached by an unsafe route is a bug you have chosen not to see.

What to Actually Build First

If you have no evals today, resist the urge to buy a platform. Build in this order.

Week one: assertions on real failures. Collect 20 to 50 cases from production traces where the system did something wrong. Write deterministic checks for whatever is deterministic - schema validity, required fields, forbidden strings, latency. Wire it into CI so it fails the build.

Week two to four: add a judge for the open-ended part. Pick the dimension that matters most - faithfulness for RAG, task success for agents, tone for a customer-facing assistant. Write a rubric, hand-label 100 rows, and measure whether the judge agrees before letting the score gate anything.

Month two: online sampling. Score a slice of live traffic with reference-free evaluators and alert on movement. This is where a hosted platform starts paying for itself, since trace collection, sampling and alerting are tedious to build.

Ongoing: grow the set from bugs. Every incident adds a row. This habit matters more than any tool choice.

On tooling: for maximum control and minimum lock-in, promptfoo or DeepEval plus Ragas metrics in your own CI is a complete stack at zero licence cost. For tracing and evals in one self-hosted place, Langfuse. Already on LangChain? LangSmith removes friction. Want a hosted playground-to-production loop? Braintrust or Phoenix. Migrating off OpenAI's hosted Evals platform? promptfoo is the path OpenAI itself documents.

How This Guide Was Built

This is an editorial synthesis, not a benchmark. We ran no head-to-head tests and make no first-party performance claims. Licences, features and pricing were read from each project's own repository, documentation or pricing page in July 2026 and are cited inline. The deprecation timeline for OpenAI's hosted Evals platform comes from OpenAI's published deprecations page, and the promptfoo acquisition from promptfoo's own announcement. Claims about judge accuracy and bias come from the cited literature, not vendor marketing. We deliberately omitted adoption statistics, market-share figures and comparative benchmark scores between these tools, because no primary source we would stake the guide on supports them. Pricing and licence terms change - verify before you commit.

Conclusion

Evals are the least exciting and most load-bearing part of shipping an LLM product. Guardrails keep your worst single response off the screen; only an eval tells you this month's version is worse than last month's. The three primitives - dataset, task, scorer - have not changed, and the tooling around them in 2026 mostly competes on how pleasant it makes the loop, not on what the loop is.

Start small and real. A few dozen production failures, a deterministic CI gate, and one calibrated judge on the dimension you care about will outperform an elaborate platform nobody wired into a build. The teams that measure quality are not the ones with the best tools. They are the ones whose eval runs whether or not anyone remembers to run it.

This is an editorial synthesis of documentation, public data, and community reports; see our [methodology](/methodology). Verify current details with each project.