Token Costs Now Rival Developer Salaries. The Fix Is Architectural.

Somewhere in your organization, an AI agent is reading a four-thousand-line schema dump it did not need, to generate a query it will get wrong on the first pass, then retry. Every line of that is billable, and the bill is no longer a rounding error.

The bill is bigger than the sticker price

For most of the last two years, the AI cost conversation has been about the price of a token, and that price has been falling fast. The trap is that falling unit prices have coincided with exploding total bills.

Forbes recently profiled a culture of "tokenmaxxing," where companies celebrate engineers for how much they spend on inference. One Databricks engineer burned through more than $7,000 in tokens in a two-week stretch, and leadership had the room applaud. An executive at another AI startup put the new expectation bluntly: if a half-million-dollar engineer is not also consuming a quarter-million dollars in tokens, something is wrong.

Gartner now projects that spending on AI coding agents is on track to rival, and in some cases exceed, the average developer's salary within a couple of years. Its research already finds roughly a quarter of technology leaders spending $200 to $500 per developer per month, and about 6% spending more than $2,000. Asia Financial reports cases where token spend passes the cost of the employee within a month or two of heavy use.

The price you are paying today is below the true cost of the compute, and that should worry a CFO more than the headline figures. Frontier labs are pricing inference below what it costs to serve, funded by venture capital and hyperscaler cross-subsidies, to capture the market. A senior Gartner analyst put it plainly: the largest labs are not making money right now, and as they push toward profitability, the efficiency gains will accrue to them, not flow through to you as lower prices. Arize, analyzing the model providers' economics, reaches the same conclusion: the subsidy era is ending, and the metered bill is what remains when it does.

So the trend line is the opposite of reassuring. Per-token prices are falling, the true cost per token is higher than the sticker, and total consumption is climbing faster than either. Agentic workflows are the accelerant. A chatbot answers once. An agent reasons in a loop, calls tools, checks its own work, and retries, consuming orders of magnitude more tokens per task. Cheaper tokens invite more spending, which is the Jevons paradox. And most teams cannot even see it clearly: in a CloudZero survey, only about half of organizations said they could confidently evaluate the return on their AI spend.

Optimize how many tokens you spend

Almost every cost-control effort in the market right now aims at the wrong variable. Buy cheaper models. Negotiate volume discounts. Wait for prices to drop. All of that optimizes the price of a token. The far bigger lever is the number of tokens you spend at all, and that number is set by your architecture, not your vendor's price sheet.

Token spend at scale comes from three places:

1. Invocations. How often a model is called, including the many calls a deterministic system could have answered without any inference.
2. Context. How much the model has to read on every single request: schema dumps, documentation, sample SQL, an ever-growing system prompt.
3. Redundancy. How often the same reasoning is repeated: identical questions asked in different words, regenerated SQL, repair loops.

None of these three is a model-pricing problem. They are engineering problems. So use language models only where probabilistic reasoning is required, and deterministic systems everywhere else. Every token you avoid is an engineering win.

This is where the semantic layer cuts cost, before you turn on a single optimization feature like caching or model routing.

What the semantic layer takes off the bill

A semantic layer is usually sold as a governance or consistency tool. In an agentic world, it is also the cheapest structural intervention available, because it attacks all three drivers of spend at the source.

It turns invention into retrieval. Without governed semantics, an agent has to discover your data model through prompt engineering. It infers which tables relate, guesses at foreign keys, and invents metric definitions. That is an open-ended reasoning problem, and open-ended reasoning is exactly what burns the most tokens and produces the least reliable output. A semantic layer converts it into a constrained execution problem. The agent looks the model up rather than learning it from a prompt: business entities, approved joins, certified measures, aggregation rules, security policies. Reasoning shifts from invention to retrieval, and that shift is where both the cost saving and the accuracy gain come from.

It shrinks the prompt. Without semantics, every request tends to carry the whole data model as context, because you never know in advance which part the agent will need. So the prompt accumulates: the full schema, column documentation, dozens of sample queries, join hints, caveats, tribal notes from analysts who left two years ago. With a semantic layer, the agent retrieves only the objects the current question touches. The measure it needs, the dimension, the relevant time grain, the applicable policy. Context engineering stops being a curation chore that grows without bound. The semantic layer is the context.

It collapses the repair loop. Bad SQL is expensive, and it is expensive twice. The first pass generates a query, executes it, hits an error, analyzes the error, regenerates, and repeats. Every lap through that loop burns tokens on the model side and warehouse time on the compute side. Governed semantics prevent most of those loops from ever starting, because the agent cannot invent a join or guess a foreign key that the semantic layer does not sanction. The search space of possible queries collapses to the ones that are valid by construction.

Some of the savings never touch the model bill at all. Incorrect SQL drives warehouse spend on its own: unnecessary and duplicate joins, Cartesian products, fan and chasm traps, incorrect aggregation, and needless full-table scans. Semantic guardrails remove entire classes of these queries before they run. Those savings accrue independently of what you pay per token, and they show up on a completely different line item.

Finally, the layer defines each concept once, for every consumer. Revenue gets defined a single time, certified, versioned, and secured, and then every consumer uses that definition: the agent, the BI tool, the notebook, the internal application. Change it once and everything updates, with no prompt rewrites and no duplicate context engineering per surface. Review effort shifts too. Analysts interpret results instead of re-validating tables, joins, and metrics on every request. And when the person who authored the logic leaves, the definition stays behind instead of walking out the door.

None of the above requires a caching tier or a routing policy. These are properties of the architecture itself, which is where the largest and most durable cost reductions come from.

After that, the optimizations compound

Once the semantic layer has removed the unnecessary reasoning, the familiar optimization techniques finally have something clean to work on, and they compound rather than compete.

Caching applies at three levels. A question cache recognizes that "revenue by region last quarter," "regional sales last quarter," and "quarterly revenue by geography" are one normalized request, and answers a repeat with zero inference. A SQL cache skips regeneration for a known request. A result cache, where freshness policy allows, skips both inference and the warehouse entirely.

Model routing sends each task to the smallest capable model and escalates only when confidence is low. Intent classification, metric lookup, and metadata retrieval do not need a frontier model. Most analytical requests never escalate at all, which makes a premium model the exception.

Deterministic validation checks everything checkable before inference, so failures become predictable rather than probabilistic, and estimates scan cost against policy before execution, so there are no surprise warehouse bills. Semantic query planning goes further still, letting the model emit a compact plan that deterministic services expand into dialect-correct SQL, so frequent business questions become certified assets where inference approaches zero.

Every one of these matters, but sequence matters more. Routing and caching optimize work. A semantic layer removes work that should never have existed.

Where to start

The instinct in a cost crunch is to reach for the cheapest model. The better move is to delegate the most work to systems that cost no tokens at all. The cheapest agent is the one that hands the largest share of its work to deterministic systems and calls a model only for the step that needs probabilistic reasoning.

That is what a semantic layer delivers before any optimization feature is switched on, and it is the foundation the compounding optimizations build on. At Coginiti, this is the architecture we operationalize: governed semantics that turn an open-ended reasoning problem into a constrained execution problem.

If you are trying to bring an AI bill down, audit the three drivers before you renegotiate a rate. Count the invocations a deterministic system could have answered, measure how much schema you are shipping on every request, and log how many queries take more than one pass.