Token Costs Now Rival Developer Salaries. The Fix Is Architectural.

Somewhere in your organization, an AI agent is reading a four-thousand-line schema dump it did not need, to generate a query it will get wrong on the first pass, then retry. Every line of that is billable. And the bill is no longer a rounding error.

The cost problem is real, and it is bigger than the sticker price

For most of the last two years, the AI cost conversation has been about the price of a token, and that price has been falling fast. The trap is that falling unit prices have coincided with exploding total bills.

The numbers coming out of the field are genuinely startling. Forbes recently profiled a culture of "tokenmaxxing," where companies celebrate engineers for how much they spend on inference. One Databricks engineer burned through more than $7,000 in tokens in a two-week stretch, and leadership had the room applaud. An executive at another AI startup put the new expectation bluntly: if a half-million-dollar engineer is not also consuming a quarter-million dollars in tokens, something is wrong.

That is not a fringe anecdote anymore. Gartner now projects that spending on AI coding agents is on track to rival, and in some cases exceed, the average developer's salary within a couple of years. Its research already finds roughly a quarter of technology leaders spending $200 to $500 per developer per month, and about 6% spending more than $2,000. Analysts tracking the same trend describe cases where token spend passes the cost of the employee within a month or two of heavy use.

Here is the part that should worry a CFO more than the headline figures. The price you are paying today is not the true cost of the compute. Frontier labs are pricing inference below what it costs to serve, funded by venture capital and hyperscaler cross-subsidies, to capture the market. A senior Gartner analyst put it plainly: the largest labs are not making money right now, and as they push toward profitability, the efficiency gains will accrue to them, not flow through to you as lower prices. Independent analyses of the model providers reach the same conclusion: the subsidy era is ending, and the metered bill is what remains when it does.

So the trend line is the opposite of reassuring. Per-token prices are falling, the true cost per token is higher than the sticker, and total consumption is climbing faster than either. Agentic workflows are the accelerant. A chatbot answers once. An agent reasons in a loop, calls tools, checks its own work, and retries, consuming orders of magnitude more tokens per task. Cheaper tokens do not reduce spend, they invite more of it. It is the Jevons paradox playing out in real time. And most teams cannot even see it clearly: in one industry survey, only about half of organizations said they could confidently evaluate the return on their AI spend.

Stop optimizing the price of a token. Optimize how many you spend.

Almost every cost-control effort in the market right now aims at the wrong variable. Buy cheaper models. Negotiate volume discounts. Wait for prices to drop. All of that optimizes the price of a token. The far bigger lever is the number of tokens you spend at all, and that number is set by your architecture, not your vendor's price sheet.

Token spend at scale comes from three places:

1. Invocations. How often a model is called, including the many calls a deterministic system could have answered without any inference.
2. Context. How much the model has to read on every single request: schema dumps, documentation, sample SQL, an ever-growing system prompt.
3. Redundancy. How often the same reasoning is repeated: identical questions asked in different words, regenerated SQL, repair loops.

Notice that none of these three is a model-pricing problem. They are engineering problems. The design principle that follows is simple to state: use language models only where probabilistic reasoning is genuinely required, and use deterministic systems everywhere else. Every token you avoid is not a cheaper inference. It is better engineering.

This is where the semantic layer earns its place in the cost conversation, and it does so before you turn on a single optimization feature like caching or model routing.

The semantic layer is a cost lever before it is anything else

A semantic layer is usually sold as a governance or consistency tool. In an agentic world, it is also the cheapest structural intervention available, because it attacks all three drivers of spend at the source.

It turns invention into retrieval. Without governed semantics, an agent has to discover your data model through prompt engineering. It infers which tables relate, guesses at foreign keys, and invents metric definitions. That is an open-ended reasoning problem, and open-ended reasoning is exactly what burns the most tokens and produces the least reliable output. A semantic layer converts it into a constrained execution problem. The agent does not learn your model from a prompt, it looks the model up: business entities, approved joins, certified measures, aggregation rules, security policies. Reasoning shifts from invention to retrieval, and that shift is where both the cost saving and the accuracy gain come from.

It shrinks the prompt. Without semantics, every request tends to carry the whole data model as context, because you never know in advance which part the agent will need. So the prompt accumulates: the full schema, column documentation, dozens of sample queries, join hints, caveats, tribal notes from analysts who left two years ago. With a semantic layer, the agent retrieves only the objects the current question actually touches. The measure it needs, the dimension, the relevant time grain, the applicable policy. Context engineering stops being a curation chore that grows without bound. The semantic layer is the context.

It collapses the repair loop. Bad SQL is expensive, and it is expensive twice. The first pass generates a query, executes it, hits an error, analyzes the error, regenerates, and repeats. Every lap through that loop burns tokens on the model side and warehouse time on the compute side. Governed semantics prevent most of those loops from ever starting, because the agent cannot invent a join or guess a foreign key that the semantic layer does not sanction. The search space of possible queries collapses to the ones that are valid by construction.

It lands savings directly on the warehouse bill. This one has nothing to do with model pricing, which is why it is easy to overlook. Incorrect SQL drives warehouse spend on its own: unnecessary and duplicate joins, Cartesian products, fan and chasm traps, incorrect aggregation, and needless full-table scans. Semantic guardrails remove entire classes of these queries before they run. Those savings accrue independently of what you pay per token, and they show up on a completely different line item.

It defines each concept once, for every consumer. Revenue gets defined a single time, certified, versioned, and secured, and then every consumer uses that definition: the agent, the BI tool, the notebook, the internal application. Change it once and everything updates, with no prompt rewrites and no duplicate context engineering per surface. Review effort shifts too. Analysts interpret results instead of re-validating tables, joins, and metrics on every request. And when the person who authored the logic leaves, the definition stays behind instead of walking out the door.

None of the above requires a caching tier or a routing policy. These are properties of the architecture itself. That is the point worth sitting with: the largest, most durable cost reductions come from the structure of the system, not from the optimizations you layer on top of it.

Then, and only then, the optimizations compound

Once the semantic layer has removed the unnecessary reasoning, the familiar optimization techniques finally have something clean to work on, and they compound rather than compete.

Caching applies at three levels. A question cache recognizes that "revenue by region last quarter," "regional sales last quarter," and "quarterly revenue by geography" are one normalized request, and answers a repeat with zero inference. A SQL cache skips regeneration for a known request. A result cache, where freshness policy allows, skips both inference and the warehouse entirely.

Model routing sends each task to the smallest capable model and escalates only when confidence is low. Intent classification, metric lookup, and metadata retrieval do not need a frontier model. Premium models become the exception, not the default, and most analytical requests never escalate at all.

Deterministic validation checks everything checkable before inference, so failures become predictable rather than probabilistic, and estimates scan cost against policy before execution, so there are no surprise warehouse bills. Semantic query planning goes further still, letting the model emit a compact plan that deterministic services expand into dialect-correct SQL, so frequent business questions become certified assets where inference approaches zero.

Every one of these matters. But sequence matters more. Routing and caching optimize work. A semantic layer removes work that should never have existed. Optimizing a query you should not be running is a slower way to pay for a mistake.

The most cost-effective agent

The instinct in a cost crunch is to reach for the cheapest model. The better move is to delegate the most work to systems that cost no tokens at all. The most cost-effective agent is not the one running the cheapest model. It is the one that hands the largest share of its work to deterministic systems, and calls a model only for the one step that genuinely needs probabilistic reasoning.

That is what a semantic layer delivers today, before any optimization feature is switched on, and it is the foundation the compounding optimizations build on. At Coginiti, this is the architecture we operationalize: governed semantics that turn an open-ended reasoning problem into a constrained execution problem, so the tokens you never spend become the cheapest ones on the bill.

The price of a token is the vendor's problem to solve. How many you spend is yours. Architecture is how you win that one.