I have set up evals for two very different systems. The first was the chat agent in Spotter Studio, a now-retired tool that helped YouTube creators come up with video ideas and titles. The evals there covered routing and behavior: did a request go to the right part of the agent, and did the agent then do what it was supposed to do. The second was a retrieval system at a nonprofit, where the question was whether semantic search over social posts and media returned the right items, and whether a new embedding model was actually better than the old one.

Both used Braintrust. I have also used LangSmith and Datadog's LLM Observability. This post covers what I would set up again, the mistakes worth skipping, and how the three platforms compare today.

What an eval is made of

Every eval platform uses the same four parts, with different names:

• Dataset: inputs, and where possible the expected output or the facts a good output must contain.
• Task: the code under test. It takes one input and returns an output: a route, a tool call, an answer, a ranked list.
• Scorers: functions that turn an output into a number. Some are plain code (exact match, recall@k, "did it call the tool with a valid argument"). Some are an LLM judge with a rubric. Some are a human.
• Experiment: one run of the task over the dataset with a fixed version of the code, prompt and model, stored so it can be compared with the next run.

The dataset and the scorers are the part that takes real work. The platform's job is to make experiments cheap to run and easy to compare.

Offline evals for an agent

Offline evals run before a change ships, against a fixed dataset. For a chat agent I split them into three layers, cheapest first.

Routing evals

A creator-facing chat agent gets many kinds of requests: brainstorm video ideas for a channel, rewrite a title, explain why one title might work better than another, or something the agent should not attempt at all. A router decides which flow, tool or sub-agent handles each one. If routing is wrong, nothing downstream can fix it, so this is the first thing to test.

Routing evals are the easiest evals you will ever write. The dataset is a list of user messages, each labeled with the route it should take. The scorer is exact match. A case looks something like this (illustrative, not a real Spotter case):

{
  "input": "my last video on budget travel flopped, give me 5 better title options",
  "expected": { "route": "title_generation" },
  "metadata": { "source": "synthetic", "tricky": "about a past video, but asks for titles" }
}

Three things make these evals useful:

• Collect cases from production, not from imagination. The messages that break a router are the ambiguous ones: a request for ideas that also asks for titles, or a follow-up that only makes sense given the previous turn. You find those in real conversations.
• Look at the confusion matrix, not only the accuracy. One number going from 91% to 93% tells you little. Knowing that "title rewrite" keeps getting sent to "idea generation" tells you what to fix.
• Run them on every change. The scoring is deterministic, but the router is a model call, so it still costs something and can vary between runs. Include the previous turns for follow-up cases, and repeat borderline cases a few times before making them a blocking check on pull requests.

Behavior evals

Once the request reaches the right place, the question becomes whether the agent behaved well: did the ideas fit the channel, were the titles in the requested style and length, did it ask a clarifying question when it should have, did it avoid promising things the product could not do.

Most of this needs an LLM judge. The practices that make a judge useful:

• One rubric question per scorer. "Is every title under 70 characters" is plain code. "Does each idea fit the channel description" is a judge. A single judge asked to rate "overall quality" produces a number nobody can act on.
• Pin and version the judge. The judge's prompt and model are part of the experiment. If they change, the scores are no longer comparable with last week's.
• Calibrate the judge against people. Label a few dozen outputs yourself, run the judge on the same outputs, and read the disagreements. You will usually find the rubric is ambiguous, not that the judge is bad.
• Prefer pairwise comparisons for subjective output. "Is version B's set of titles better than version A's for this channel" is easier to judge reliably than an absolute score out of 10.

Trajectory checks

For an agent that calls tools, the final answer is not the only thing that matters. Useful code scorers check the path: the tool that was called, whether its arguments were valid, how many steps it took, and whether it did anything it must never do. These are cheap and catch a surprising number of regressions after a prompt change.

What a retrieval eval taught me about eval hygiene

The nonprofit's retrieval eval was not an agent eval, but it taught me the most about keeping an eval honest. The existing setup was a test in the repo's test runner: 20 questions with hand-labeled correct answers, a corpus of a few thousand text posts and media items, and a GitHub Action that posted precision, recall and F1 to the pull request when a label was added.

It had a quiet bug in its design. The runner retrieved exactly as many items as each question expected, so precision, recall and F1 were always the same number. That is mathematically expected when k equals the number of relevant items, so nothing was broken in the code. But it meant the three metrics could not move independently, and the eval said nothing about what happens with a fixed retrieval budget. Before trusting any eval, check that each metric can actually disagree with the others.

What I changed, and would do again:

• Two jobs, two tools. I kept the test in the test runner as the hard gate: below a set score, the pull request fails. Braintrust ran alongside it for everything the gate is bad at: side-by-side comparison of embedding models, per-question drill-down, and history across pull requests. A CI gate should be simple and boring. A place to explore results should be rich. Keeping them separate meant neither had to compromise.
• Put the experiment's identity in its name and metadata. Each run was named from the provider, model and embedding dimension, with metadata for the corpus type and k. Comparing "model A at 1024 dimensions, text only" with "model B, images included" is then a filter, not an archaeology project.
• Cache the expensive part. Embedding the whole corpus is slow and costs money. Caching the embeddings meant reruns only redid retrieval and scoring, which made iterating on the questions and scorers take seconds instead of minutes.
• Drill into single cases. The per-question view showed that text posts were retrieved reasonably well while media items were found much less often. One likely cause was easy to spot once visible: media had been embedded from its text metadata only, not from the images. An average score would never have shown that.

The scorers went through a lesson of their own. I started with about fourteen: precision, recall and F1 for each item type, NDCG, MRR, and similarity diagnostics such as the average similarity of correct and incorrect hits and the gap between them. One of them could go negative, and Braintrust expects scores between 0 and 1, so I clamped it: a workaround that throws information away, so keep the raw value somewhere as a diagnostic. Within a day I cut the list to eight: recall and mean average precision for each item type, hit rate for media, the average similarity of correct hits for each item type, and one combined score. Fourteen numbers per run meant nobody looked at any of them. Eight, each tied to a decision, got read.

Online evals

Offline evals only cover the cases you thought to collect. Online evals score real production traffic, after the fact, and they are how the dataset keeps growing.

The pattern that works:

• Run cheap code checks on all traffic and LLM judges on a sample. Formatting, length, refusals, tool errors and latency can be checked on every trace. How much to send to a judge depends on volume, how rare the failures you care about are, and which segments matter most. Sample more heavily where failures are rare or costly.
• Score asynchronously. Online scoring must never add latency to the user's request.
• Alert on the trend, not on single traces, except for severe failures such as leaked data or unsafe output, which deserve an alert every time. For everything else, a drop in the daily share of conversations the judge rates as on-topic is a signal; one bad trace is not.
• Close the loop. The most valuable output of online evals is new offline test cases. When a production trace fails, add it to the dataset with the correct label, and the next change is tested against it.
• Mind the data. Production traces contain user data. Check what the platform stores, for how long, and where, before turning on full capture, and redact before copying production traces into a dataset.

Braintrust vs LangSmith vs Datadog

I have used all three, and Braintrust is my favorite. Apart from that preference and the Braintrust details from my own projects, what follows comes from a fresh read of each vendor's documentation and pricing page in September 2026. These products ship quickly, so check anything that matters to you against the current docs.

Braintrust

Braintrust is the one I would pick again. Its core is the experiment loop: you write an Eval() with data, a task and scorers, run it, and every run becomes an experiment you can compare against a baseline, with a per-example diff and each case marked as an improvement, a regression or a tie. The per-question view in that comparison is where the media-retrieval gap above became visible.

Other things that matter in practice:

• Scores are numbers between 0 and 1. That is a small constraint (I had to clamp a scorer that could go negative), but it keeps every scorer on the same chart scale.
• Autoevals, an open-source library of ready-made scorers (factuality, similarity, JSON validity and others), plus LLM-judge and plain-code scorers.
• An official GitHub Action that runs evals on a pull request and posts the results as a comment.
• Online scoring through rules that run scorers asynchronously on a sample of production spans or whole traces.
• Agent guidance in the docs that covers routing ("did it go to the correct model or sub-agent"), tool choice and tool arguments. It has no packaged trajectory matcher or simulated-user feature. You write those as scorers.
• No per-seat pricing, which matters when you want product managers and designers in the tool reading results.

The weaknesses: the platform is closed source, self-hosting is Enterprise-only, and scores are metered. If you run judges on a lot of production traffic, the score count is the number to watch.

LangSmith

On paper, LangSmith has the most built-in tooling for agent-specific evaluation, and its own libraries do more of the work for you. agentevals gives you trajectory matching in four modes (strict, unordered, subset, superset), openevals can simulate a user for multi-turn tests, and the docs have a dedicated guide for single-step routing evals. Online evaluators can have a weekly spend cap, which is a thoughtful feature for LLM judges on live traffic, and alerts can go to Slack or PagerDuty.

It works without LangChain, through its own SDK or OpenTelemetry, and it integrates most deeply with LangChain and LangGraph apps. For CI it offers pytest and Vitest/Jest plugins rather than a packaged GitHub Action. Pricing is per seat plus per trace, and the overage rate sits behind a calculator rather than on the page. Having used both, I still prefer Braintrust.

Datadog LLM Observability

Datadog's product, which it is now also calling Agent Observability, has grown a full eval toolset: versioned datasets, experiments with side-by-side results, LLM judges at span, trace and session level, nine judge templates including tool selection and tool-argument correctness, annotation queues (since March 2026), and a prompt registry. Since June 2026 it can also run DeepEval and Pydantic Evals metrics unchanged.

Its real advantage is everything around the evals. LLM traces sit next to your APM traces, logs and real-user monitoring, and eval results can drive the same monitors your on-call rotation already uses. Its pricing is also the simplest of the three: you pay per LLM call span, and tool and retrieval spans are free. There is no separate evaluation fee, but the LLM calls a judge makes count as billable spans.

Where it is weaker: it started as an observability product, and most of its offline tooling (datasets, experiments, annotation queues) is newer than the other two platforms'. I found no packaged eval action for CI in the docs, so pass/fail thresholds in a pipeline need your own integration code around the SDK. It is SaaS only.

Feature comparison

	Braintrust	LangSmith	Datadog
Framework-agnostic SDKs	Python, TS, Go, Java, Ruby, C#	Python, TS	Python, Node, Java
OpenTelemetry ingest	Yes	Yes	Yes
Datasets from production traces	Yes	Yes	Yes
Experiment comparison	Per-example diff, regressions flagged	Diff mode for two runs	Side by side
Code and LLM-judge scorers	Yes, scores in [0, 1]	Yes, any range	Yes, typed outputs
Trajectory / tool-call evals	Custom scorers	agentevals, four match modes	Tool selection and argument templates
Simulated users	Not built in	Yes (openevals)	Not built in
Online evals with sampling	Yes	Yes, with spend caps	Yes
CI	Official GitHub Action	pytest / Vitest plugins	Your own code around the SDK
Human review queues	Yes	Yes, including pairwise	Yes
Self-hosting	Enterprise only	Enterprise add-on	No

Pricing, as of September 2026

	Free tier	Named paid plan	What you pay for
Braintrust	1 GB of data, 10k scores a month, 14-day retention, unlimited users	Pro, $249 a month: 5 GB, 50k scores, 30-day retention	Data and scores (the free tier can also pay for overage), no seat charge
LangSmith	1 seat, 5k traces a month, 14-day retention	Plus, $39 per seat a month: 10k traces	Seats and traces; overage rate not published as text
Datadog	40k LLM spans a month, 15-day retention	$160 a month for up to 100k LLM spans	LLM call spans only, including the judge's own calls

One pricing detail to watch in LangSmith: experiments and scored traces default to extended retention, which costs more than base traces unless you turn it off. The free tiers are all large enough to find out whether evals change how your team works, which is the only question that matters in the first month.

How I rank them

Criteria first, because the ranking depends on them. I am ranking for a small team shipping an agent, where the scarce resource is engineers' attention and the goal is to know whether a change made the agent better or worse before it ships.

1. Braintrust. In my experience, the shortest path from "I changed a prompt" to "here are the cases that got worse." Official CI support, no seat pricing, and the experiment comparison I liked working in most.
2. LangSmith. On paper, the most built-in tooling for agent-specific evals: trajectories, simulated users, routing guides. Choose it first if you are on LangGraph, or if multi-turn simulation is central to what you need to test.
3. Datadog. The right answer if your company already runs on Datadog and wants LLM quality in the same place as latency, errors and on-call. Per-LLM-call pricing with no separate eval fee makes the bill easy to predict.

Two self-hostable options are worth knowing about even though I have not used them in production. Langfuse is MIT-licensed at its core, can be self-hosted for free, and has an official GitHub Action for experiments. It is a strong option when traces must stay on your infrastructure, as long as the judge model runs there too. Arize Phoenix is free to self-host under the Elastic License and has good tool-call and trajectory metrics. You can score production traces with its SDK, but managed continuous online evals and alerting come with Arize's commercial product.

What I would set up first

Whichever platform you pick, the order matters more than the tool:

1. Fifty routing cases taken from real conversations (redacted), scored by exact match, run in CI.
2. Three to five behavior scorers, each answering one question, with the judge prompt versioned and checked against a few dozen human labels.
3. Code checks on every production trace, and one judge on a small sample, scored asynchronously.
4. A weekly habit of turning failed production traces into new test cases.

The platform makes steps 1 to 4 faster. It cannot choose the cases for you, and the cases are where the quality comes from.