When the Model Eats the Stack, Semantics Become More Important

A recent paper from researchers at UC Berkeley and Stanford asks a question every data infrastructure roadmap should be stress-tested against: what happens when increasingly capable models absorb the functionality we have been building around them?

The paper, What Happens When the Model Eats the Stack? Rethinking the Research Agenda for Data Agents to Withstand the Bitter Lesson, starts from Richard Sutton's "bitter lesson", the observation that general methods which benefit from increased computation tend, over time, to outperform systems built around hand-engineered knowledge and task-specific heuristics.

Applied to data agents, the implication is uncomfortable. A great deal of current agent architecture exists because models are not yet capable enough on their own. We build planners, decomposers, SQL-specific agents, validators, retry loops, and layers of orchestration designed to compensate for weaknesses in the underlying model. What happens as those weaknesses disappear?

The paper shows the process is already underway. It also shows that semantic problems, unlike the engineering ones, become more prominent as models improve.

Better models are eating the agent stack

The researchers compared general-purpose coding agents with specialized, human-designed data agents using multiple generations of frontier models, across two public benchmarks. TAG-Bench tests questions that require combining exact computation with semantic reasoning and world knowledge, the kind of question that cannot be expressed in relational algebra alone. The Data Agent Benchmark is closer to enterprise reality: 54 queries across 12 datasets, nine domains, and four different database systems, grounded in a formative study of data agent workloads across six industries. Its premise is fragmentation, with inconsistent references across heterogeneous systems and information buried in unstructured text. When DAB was published in March, the best frontier model on it managed 38 percent accuracy.

With weaker models, the specialized architectures sometimes performed better. By GPT-5.6 Sol, the general coding agent substantially outperformed the specialized agents tested on both accuracy and token efficiency, with no task-specific engineering behind it.

Efficiency moved further than accuracy. On DAB, the average number of turns required to complete a query fell from 23.2 with o3 to 9.9 with GPT-5 and 6.0 with GPT-5.6 Sol, roughly a fourfold reduction across those generations. Much of the current thinking about agent-first data infrastructure assumes the opposite trajectory, in which agents become highly speculative consumers of data systems, issuing large numbers of exploratory queries and repeatedly correcting themselves. The measurements here point the other way. As models improve they need fewer steps to reach the correct result, and they spend less of those steps on debugging, verification, and recovery.

None of this proves that every specialized data agent will disappear or that agent workloads will become cheap in aggregate. But it should make platform builders cautious about betting heavily on infrastructure whose primary purpose is compensating for current model limitations. Models are improving quickly. Your architecture needs to solve problems that survive their improvement.

What the model still does not know

As the models got better at planning, computation, debugging, and tool use, a larger share of their remaining work involved something they could not reason their way through: understanding the particular data environment in which they were operating.

The agent still has to determine what data exists, which source is authoritative, what a given field means, which metric represents the business concept in the question, what the correct entity grain is, which identifiers refer to the same real-world entity, and which join path is valid. It has to know what the organization means by revenue, customer, active, or churn. These are knowledge problems rather than reasoning problems, and the knowledge belongs to your organization rather than to the model's pretraining.

The failure analysis makes the shift visible. Execution errors, broken tool calls, and malformed commands declined steadily as capability improved. For GPT-5.6 Sol, more than 60 percent of remaining failures were attributed to semantic or environmental causes: misunderstanding the requested concept, selecting the wrong source or metric, or misusing entities, identifiers, and join keys. The composition of effort moved the same direction. Even as total turns collapsed, the share spent on schema exploration rose from 16 to 25 percent. The bottleneck moved rather than disappeared.

The semantic contract

The context the model generated for itself is the most revealing part of the paper.

When the researchers had the agent write reusable instructions based on its own prior successes and failures, the instructions told it to fix a set of decisions before computing anything: eligible population, entity grain, unique key, join keys, date bounds, metric, grouping, ranking, tie policy, and output shape. The authors call this establishing a semantic contract.

That list is a nearly exact inventory of what the strongest model was still getting wrong, and the agent, given the chance to instruct itself, reached for it unprompted.

It is also, almost line for line, the job of a modern semantic system. Semantic layers have often been described in terms of query generation: define measures and dimensions so that BI tools, and now agents, can produce the right SQL. That framing undersells the durable value, which is the explicit contract for what the SQL is supposed to mean. As model SQL generation improves, the contract matters more, not less.

Persistent semantic context

The authors describe the missing ingredient as persistent semantic context: an explicit external representation of knowledge about the data environment, built once, maintained over time, and reused across many agent interactions. Their examples include canonical metric definitions, authoritative data sources, entity relationships, organizational conventions, historical discoveries, and user preferences.

A model can learn SQL during training. It cannot learn that your company calculates Net Revenue using a particular set of exclusions from a specific governed source, because that definition may be private, newly created, frequently revised, or different from the one another business unit uses. That knowledge has to live outside the model and be available when the agent needs it.

The researchers tested this directly, constructing context offline from previous agent trajectories and supplying it to GPT-5.6 Sol on held-out tasks. The best-performing context improved accuracy by 19 percentage points over the no-context baseline.

That result hands skeptics a reasonable objection. If an agent can author a usable semantic representation from a handful of prior traces and gain 19 points, why buy anything? The answer is in what the artifact lacks. It has no owner, no version history, no scope boundary, no consistency guarantee, and no audit trail. It was frozen at build time and appended to a prompt. Nobody approved it, nothing invalidates it when the underlying definitions change, and nobody can explain six months later why a number moved. Generating context is getting cheap. Governing it is not, and that is where the durable engineering problem sits.

The experiment carries a second warning. More context was not automatically better. A schema-focused variant reduced the exploration the agent had to perform, yet accuracy fell modestly, which the authors attribute to over-reliance on the supplied context and overfitting to a small development set. Agents need the semantic context that fits the task in front of them.

From semantic layer to semantic intelligence

This is why we think the semantic layer is better understood as one component of a larger semantic intelligence architecture.

A conventional semantic model captures defined semantics: measures, dimensions, entities, relationships, grain, join paths, and governed business definitions. Agents need more than that. They draw on enterprise context: catalog descriptions, ownership, documentation, lineage, transformation logic, and approved analytical assets. They benefit from learned context, the validated discoveries of previous executions, known join hazards, useful examples, and entity-resolution decisions that were expensive to make the first time. And every execution carries task context: the current question, the slice of the semantic graph relevant to it, and intermediate results that live and die with the request.

The architecture this implies is a large governed corpus of persistent semantic knowledge from which small, task-specific contexts are compiled on demand. The paper's own schema-context result shows what happens when you dump a catalog into a context window instead. The semantic model becomes the authoritative source from which reliable context is generated.

Governance becomes more important as models improve

If the model handles more of the planning and execution, the surrounding enterprise infrastructure shifts toward a different set of questions. Which definition should the agent trust? Which data product is authoritative, and who approved that metric? Which version was valid when this analysis ran? Should every agent immediately observe a changed business definition, and what happens when two teams legitimately use different ones? How do we know the context is still correct after the schema changed underneath it?

The authors call this semantic consistency and draw the analogy to consistency models in databases, sketching a design space that includes strong and eventual semantic consistency, query-triggered revalidation, multiple scopes or namespaces, and incremental maintenance of semantic artifacts. This part of the paper is explicitly a research agenda rather than a result.

It is also, for anyone who has run a governed semantic program, a familiar requirements list. Scope, versioning, propagation, conflict resolution, freshness, and audit have been product requirements for years. What the paper contributes is the measurement that shows why they became load-bearing for agents specifically. A semantic definition earns its keep when it is current, governed, attributable, versioned, and delivered to the right consumer.

Why structured semantics still matter

The paper does not argue that persistent context must take the form of a traditional semantic model. The authors deliberately leave open whether future systems will rely on natural-language files, knowledge graphs, vector indexes, or some combination, and that is the right way to hold the question. Different kinds of semantic knowledge deserve different representations. Free-form text documents organizational convention well. Vector indexes support retrieval. Historical traces preserve operational knowledge. Graphs represent relationships.

Authoritative enterprise semantics, though, benefit from structure. There is a real difference between an agent remembering that customers are usually joined to orders on customer_id and a governed model that encodes the Customer entity, its canonical key, the relationship to Order, the permitted join path, the applicable cardinality, and the metrics built on top of it. Structure is what gives you explicit dependencies, provenance, testability, versioning, ownership, and deterministic execution. Those properties are optional in a demo and mandatory in a business process.

What the paper shows, and what it does not

The evidence deserves the same scrutiny we would want applied to our own claims.

The scale is small. The held-out evaluation set was 42 tasks, so a 19-point delta rests on roughly eight questions, and the failure taxonomy draws on a few dozen traces. Every model tested came from a single vendor family, run at low reasoning effort. One of the specialized baselines could not be fully reproduced because parts of its inference pipeline are not public. And the failure classification was performed by the same model whose failures were being classified, which is not a neutral instrument for a category like misinterpreting the task.

The experiments support four things: that better models are more accurate and more efficient at data tasks, that sophisticated reasoning scaffolds are losing value, that most remaining failures are semantic or environmental, and that persistent context can substantially improve accuracy. They do not show that a single general model replaces every agent architecture, that agent query volume stops being an infrastructure concern, or what the ideal representation of semantic context is. They certainly do not demonstrate persistent semantic context running in production across an enterprise with thousands of tables and years of history. The authors are direct about this, naming context construction, storage, retrieval, compression, lifecycle management, and semantic consistency as open problems.

Those open problems are where we think the most important work in enterprise data infrastructure is going to happen.

What the model cannot absorb

The bitter lesson applies to us as much as to anyone. Any layer whose main purpose is compensating for model weakness has a shelf life, and orchestration, prompt scaffolding, task decomposition, and retry logic are all in that category. The honest exercise for any vendor is to audit which side of that line each capability sits on.

What does not get absorbed is meaning that belongs to the organization rather than the model. The model can write the query without knowing what revenue means here, or join two tables without knowing whether they represent the same entity at compatible grains. It can calculate a metric that Finance never approved.

Coginiti was built for that half of the problem. Semantic definitions are developed, tested, and executed against more than twenty data platforms, with the version history and governance that make an answer defensible six months after it was given, and Coginiti Forge and Coginiti Guide operate against that layer rather than rediscovering the environment on every prompt.

The semantic layer defines meaning. A semantic intelligence platform manages that meaning as persistent, governed, operational context for humans and agents alike.

---

Further reading

• The paper: What Happens When the Model Eats the Stack?
• TAG-Bench: paper and benchmark code
• Data Agent Benchmark: paper and benchmark code
• Richard Sutton, The Bitter Lesson