The first time we ran our eval suite against the analytics agent, it told us the agent was failing well over half its questions. That number turned out to say more about the suite than the agent. When we read through the failures one at a time, most of them weren’t the agent getting things wrong. They were our own scorers calling good answers bad.
‍

Some of those were ordinary bugs, the kind you fix and forget. The interesting ones were cases where an LLM judge was scoring a question that was already settled. The agent would return the correct number, our data check would confirm it against the reference query, and then the judge would knock the case down anyway because the agent phrased the answer differently than we had it written. We already knew the answer was right. We were paying a slow, expensive model to second guess a verdict a deterministic check had handed us for free. For a question like that we never cared how the answer read, only whether the one number was right, and that is exactly what the deterministic check had already settled. The judge was the wrong tool for the job.
‍

I think this mistake has a general shape, and it runs both ways. Teams reach for a judge when they already have a real answer to check against, and they lean on rigid matching for things that don’t have a single correct answer. Most of the trouble in an eval suite comes from putting a scorer where it doesn’t belong.

‍

Our suite settled into three scorers, and most of this post is about keeping each one in its lane. 
‍

1. A tolerant deterministic comparison decides whether the numbers are right. 
2. A deterministic check decides whether the agent honored the hard constraints on its path. 
3. A judge handles the prose, the one place with no answer to compare against.
‍

The first two are cheap and run on every case; only the judge costs real money, and only the judge can be wrong about its own verdict.

‍

‍

‍

When there’s a ground truth, you don’t need a model to grade the answer. If the question is “what was revenue last quarter,” there’s a number. Run the reference query, compare, done. The comparison costs nothing, gives the same verdict every time, and has no opinion about the answer.
‍

The work is in the comparison, not the grading. Naive equality fails good answers all the time. The semantic layer hands back a column named order__region where the golden calls it region. The rows arrive in a different order, or with an extra column the agent added for its own analysis. The agent aliases a value total_revenue where the reference says revenue. Audits of text-to-SQL benchmarks keep finding the same imbalance, that rigid matching rejects correct answers noticeably more often than it waves wrong ones through, which means a suite built on naive matching will report your agent as worse than it is. The rarer failure runs the other way, a structurally wrong query that happens to return the right rows on this particular dataset.
‍

Both problems have boring fixes. Normalize dates, ignore column order, project down to the columns you care about, compare numbers within a tolerance. For the lucky query, run the reference against more than one dataset and most of the luck disappears. It’s bounded work, and it’s ordinary deterministic software, not a model. You do it once and then you trust it. None of it is a reason to bring in a judge, because a judge doesn’t repair a brittle comparison. It just swaps a problem you can see for one you can’t.

There’s a tempting middle category here, whether the agent took the right path to the answer. Part of it is real and deterministic. If a case requires the semantic layer, assert that the tool got called. If a tool is off limits, assert that it didn’t. Those are hard constraints with actual ground truth, and they’re free to check.
‍

Past that, I don’t think you should fail an agent for taking a quirky route. If the answer is right and the constraints held, not finding the path you imagined isn’t a bug, and a pass-fail check on path elegance is just your opinion wearing a rubric.
‍

But the path is not noise, and efficiency is worth taking seriously, because it’s one of the more useful diagnostics the suite produces. Tool calls, tokens, seconds, those are counters, and when an agent burns eight calls discovering a metric that should take two, that’s often a sign of something you control rather than the agent being dumb. Bad tool descriptions are the first thing to rule out. Rewriting them alone measurably improves which tools an agent reaches for, as both Anthropic's guidance on writing tools for agents and recent research on rewriting tool descriptions show. But they aren't the whole story. The same wandering also comes from too many overlapping tools, a tool catalog that's grown too large, or the model not noticing it already has the answer. The trace still shows you exactly which tools the agent reached for, misread, and abandoned, and the fix often lands in a tool’s description or name rather than in the agent. So track the counters on every run, watch the trend, and actually read the traces behind the outliers. None of that needs an LLM judge; it needs a person willing to read the slow traces and act on what the counters are showing.

What’s left for the judge is the stuff with genuinely no answer to compare against, and for us that’s the prose. Users read the agent’s answer, not its rows. An agent can pull the right number and then describe it wrong, hedge it into mush, or dress up a guess to sound certain, and no comparison will ever be noticed.
‍

The version of this that actually shows up in our suite is mundane. The agent runs the right query, gets the right number, and describes it with the wrong scope, a tier-filtered revenue figure written up as if it were total revenue. The data check passes because the number matches the golden, the trace is clean, and anyone acting on the sentence still gets misled. We did try to check prose deterministically first, with required-phrase assertions on the answer, and that experiment is where a chunk of our false failures came from. The agent would write June 28, 2026 where the assertion wanted 2026-06-28. There are too many correct ways to word an answer to enumerate them, and that’s really what judge territory means.
‍

You pay for that judgment, and the bill is bigger than the per-call cost. Judges have tastes. They favor answers that sound like their own, they reward length, and they change their grades when you reword the rubric. On harder, more specialized questions they agree with human experts a lot less than you’d hope. So a judge is a signal, not a source of truth. Before one gates anything, label a sample of its verdicts by hand and see how often it agrees with people. Ours doesn’t gate a release yet, and when the judge and the data disagree, we side with the data.

‍

The clean split is the right default, and there’s one refinement worth stealing. When an exact-match check fails, the failure itself deserves a second look, because rigid matching concentrates its errors in false negatives. So escalate exactly those cases to a judge with one narrow question. Are these two results equivalent? That’s a much smaller ask than open-ended quality grading, and audits show it closes most of the gap with human reviewers on ground-truth tasks. It stays cheap too, since the judge only ever sees the failures, and the failures are a small slice of the suite.
‍

That’s what fixing our suite actually amounted to. We didn’t remove the judge or learn to distrust it in general. We narrowed what we asked of each scorer. Tolerant deterministic comparison decides correctness, counters track efficiency, and judgment is saved for the one place where judgment is the only tool there is.

‍

‍

Everything above assumes one warehouse and one set of models, and that isn’t what production looks like for us. Every client has their own dbt project, their own semantic layer, and their own data, and all of them can be weird in their own ways. A pass rate against our reference fixture tells you the agent and the harness work. It tells you very little about whether the agent answers correctly on top of a specific client’s models, where the metric names are different, the dimensions are different, and the data carries whatever debt real data carries.
‍

So the suite deploys per client. The cases are derived from each client’s own dbt artifacts, their metrics and dimensions become the question pool, and the goldens are reference queries against their warehouse, hidden from the agent the same way as before. Then it runs on a schedule, because what breaks is whatever changed most recently, and client dbt models change all the time. A model gets renamed, a dimension disappears, new data lands at a grain nobody mentioned. A scheduled run catches that drift the day it happens instead of the month a user complains about it.
‍

The economics only work because of the split this whole post is about. The deterministic checks cost nothing, so every client gets the full battery on every scheduled run, scored against that client’s own pinned baseline instead of some global number. The judge stays sampled and on escalation, so the spend scales with failures rather than with the number of clients. The side outputs get more useful too. A coverage gap surfaced on a client’s suite is a specific missing dimension in their semantic model, which is a work item you can hand the data team. A wandering trace on their tools means their tool descriptions need attention. Run this way, the suite works less like a benchmark and more like monitoring, which is what an eval should grow up to be.

If there’s a ground truth, check it directly, on every change, and put the effort into making the comparison tolerant rather than outsourcing it to a model. If the check fails, let a judge ask one narrow equivalence question before you log an agent error. Hard requirements on the path are deterministic, and efficiency is a counter you watch, because a wandering trace usually indicates your tool descriptions, not your agent. The judge gets the prose, and it doesn’t get trusted until it earns that against human labels. And if you serve more than one client, run all of it against each client’s own models on a schedule, because correct on your fixture is not correct on theirs. Everything else is spending money to make your results noisier, or trusting a grade nobody ever looked at.