Part 1 of Loop Engineering for dbt Agents, a series on making AI data modeling reliable through harness design: this post makes the claim; the next ones show how to build it.
When dbt Labs benchmarked coding agents on real analytics engineering tasks, they ran two versions - a baseline agent against an agent equipped with Skills. They gave both agents the same questions and compared the output. The baseline agent handled a multi-model request in a way anyone who has tried this will recognize: it “created 6 models at once and declared victory.” Six plausible SQL files dropped in one pass. The agent equipped with dbt Labs’ skills worked the same task iteratively - and completed it every time.
“6 models at once” (generic agent, no Skills) vs “one model at a time” (same agent + dbt Skills)
If you have pointed an agent at your dbt project, you’ve probably watched some version of this. The output compiles in your head as you read it. Then you run it, and the grain is wrong, a join fans out, a metric doubles. The natural conclusion is that AI can’t do data modeling.
So what separated the two runs? Skills are bundles of prompts and scripts that read like a training manual: get a sense of the schema and the underlying data before making changes, and after building or modifying a model, look at the data again and run summary statistics. The model was the same in both runs. The written guidance was the difference.
There are two lessons here:
- The failure was process, not capability. The same model with the same tools succeeded once it was told to work iteratively and check its output against the data.
- Skills are advisory. A skill can bundle more than prose - dbt Labs’ includes a helper script for parsing run results - but everything in it is used at the agent’s discretion. You can’t control whether the agent follows it, long tasks make it drop pieces, and when something unexpected breaks, nothing guarantees the error reaches the agent or stops it from declaring done.
The durable fix is to move the checks from advice into structure: a harness. Instructions don’t disappear - they’re how the agent knows what the harness is and how to work inside it - but the loop stops depending on the agent remembering them.
Quality lives in the harness
How well AI does dbt data modeling depends more on the harness around the model than on which model you picked.
A harness is composed of two halves.
- Context in: the model can see your project’s structure, lineage, docs, and the actual data it is modeling.
- Validation out: every change runs against real checks - compile, test, build - and the errors flow back so the agent fixes its own work before you ever see it.
Loop engineering is the discipline of wiring those signals into a cycle the agent cannot skip.
“the harness”: context in, validation out, errors flow back
For a CTO the question is where the next dollar goes: a bigger model, or the harness around it. The leverage is in the harness. For a data engineer the question is whether you can trust what the agent produces. Your skepticism is warranted, and the loop is the mechanism that helps put your mind at ease.
More harness beats more model
This isn’t intuition, and it isn’t new… It was observed in papers from 2023, ancient history at AI speed. Reflexion wrapped a feedback loop around GPT-4 - generate, run the tests, reflect on the failures, retry - and hit 91% pass@1 on HumanEval, against 80% for the same model without the loop. +11 points from wiring, not weights. Self-Debug found that a model inspecting its own execution results improved its text-to-SQL accuracy with no human feedback at all, and that one model iterating on feedback can match a model generating over ten times as many candidates. Feedback substitutes for brute force.
The most telling evidence is first-party. dbt Labs built ADE-bench to compare how different models perform on real analytics engineering tasks. In their retrospective, they share an interesting finding:
the biggest gains were unlikely to come from better models, but from more context.
A vendor with every incentive to tell you to buy a bigger model measured the problem and told you to build more context. But context is only half the harness; validation is the other.
The Skills experiment from the opening is another example of this. What it measured was one slice of the harness: procedural context. Even that slice - a bundle of written guidance - fixed one of the most common failure points dbt Labs had seen. This is evidence of where agents fail by default, not a ceiling on what an engineered loop can do. Skills are the lightest weight layer of harness: guidance and helper tools the agent may or may not use. Don’t stop there - add the context layer and the validation layer, and close the loop.
dbt was accidentally built for agentic harnesses
Data modeling might look like a hard domain for agents. We think dbt is actually one of the best environments an agent can work in, for a surprising reason: dbt’s error signals are mechanical, not interpretive.
On the validation side of the harness, a failing data test doesn’t tell the agent “something is wrong somewhere.” It is a query that selects the offending rows - the agent can run it and look at them. A model contract violation identifies the column and the expected-versus-actual type before anything materializes. A compile error highlights the line where the error occurred. Each signal states the problem specifically enough that fixing it requires little diagnostic skill - the harness has already done the diagnosis.
The context side is just as compelling. A well-kept dbt project carries its business knowledge in machine-readable form: manifest.json maps every model, source, and edge in the DAG; YAML descriptions and semantic definitions sit next to the SQL they describe. The conventions dbt taught us for onboarding humans turn out to be exactly what agents need.
The upper bound
Now for some bitter truths.
Olausson et al. asked whether self-repair is a silver bullet for code generation. Their answer: no. Gains from self-repair are often modest, and the loop is bottlenecked by the model’s ability to understand what the feedback means - GPT-3.5 exploited feedback notably worse than GPT-4, and gains grew when a stronger model supplied the diagnosis. A loop doesn’t rescue a model that can’t read the error.
The dbt Labs numbers say the same thing from the other side. Skills moved overall benchmark accuracy from 56% to 58.5% - modest - with the big wins concentrated in iterative multi-model tasks. Each layer of a harness targets specific failure modes. None of them is a blanket multiplier, and nobody should sell you one.
So the defensible claim is narrower than “any model works”; specifically it’s that a good harness lowers the model-capability bar substantially. It doesn’t remove it. If weaker models are worse at interpreting feedback, the harness must need less interpretation: failing rows instead of stack traces, a named type mismatch instead of a generic build failure. The better your harness converts “something is wrong” into “this specific thing is wrong, here is the evidence,” the less model you need.
Run a different experiment
If your evidence that AI can’t do data modeling comes from pointing a chat model at SQL and judging its first answer, you’ve measured a model without a harness. That number tells you almost nothing about what a loop can do - and the loop is the cheaper thing to fix. Model capability is rented and repriced every quarter. A harness is built once, and every model you ever run inside it gets the benefit, including the better ones that ship next year.
So run the experiment that actually answers the question: give an agent your project’s context, wire the validation signals into a loop it cannot skip, and judge what comes out the other end.
That wiring is what the rest of this series builds, piece by piece: the validation ladder, project structure as context engineering, and error-message design.
Next in the series: the validation ladder - dbt gives you an ordered set of feedback signals, from free and instant (dbt parse) to expensive and definitive (a full build with tests). Post 2 shows how to sequence them so an agent climbs the ladder cheapest-first and every error feeds back into the loop. Follow us to make sure you don’t miss it.