Your AI Token Cost Is Growing. Is Your Engineering Output Growing With It?

There is a conversation happening inside engineering organizations right now that probably sounds familiar. Someone looks at the AI costs. Then they look at what the engineering organization actually shipped. And the two numbers do not seem to belong together.

Chamath Palihapitiya recently shared one particularly striking example. After asking his CTO about the company’s AI spend, he was told their token costs were doubling roughly every 45 days.

The obvious next question was: what are we getting for it?

The answer was roughly a 5% increase in downstream productivity.

That is one company’s experience, not an industry benchmark. But the question behind it is one a lot of CTOs and engineering leaders are beginning to ask:

If we are consuming dramatically more AI, why aren’t we seeing dramatically more engineering output?

The easy conclusion is that AI has reached some kind of productivity ceiling. We are not convinced that is what is happening. We think a large part of the problem is much more familiar to software engineers. It is architecture.

More Tokens Do Not Automatically Mean More Productivity

For the last couple of years, increasing AI adoption was relatively easy to celebrate. More developers using AI? Good. More prompts? Good. More tokens? Probably good. More agents running autonomously? Even better.

But eventually someone has to connect those numbers to an outcome. That is where things get interesting.

Jellyfish recently analyzed token usage and engineering output across 12,000 developers at 200 companies. The data showed that developers consuming more tokens generally did produce more work. Just not proportionally more.

Developers at the low end of token usage averaged roughly 0.77 merged pull requests per week. At the high end, that increased to 2.15. That sounds promising until you look at what it took to get there.

The median developer consumed about 7 million tokens per merged pull request. Developers in the top decile consumed roughly 69 million. That is nearly ten times the tokens for a little more than twice the throughput.

The cost difference was even harder to ignore. Jellyfish estimated that cost per merged pull request increased from $0.28 in its lowest usage tier to $89.32 in its highest. Coverage from TechCrunch helped push the same finding into the broader industry conversation: “tokenmaxxing” can make teams feel more productive than the output numbers support.

So AI was helping developers produce more. The problem was how much additional AI consumption was required for each incremental improvement.

That distinction matters. The question is no longer, “Are our developers using AI?”

It is, “What are we getting for the AI we are using?”

We Have Seen This Movie Before

None of this should feel particularly foreign to an experienced software engineer. Give a system an effectively unlimited resource and eventually something will find a way to consume it.

We learned that lesson with memory, CPU, database connections, network calls, storage, retries, and cloud infrastructure. Now we are learning it with tokens.

Consider a relatively normal coding agent. You give it a task. It reads some context, reasons about the problem, calls a tool, reads the result, and tries something.

Something fails, so it reads the error and tries again. Maybe the requirement was ambiguous, so it changes direction. Maybe it pulls additional files into context. Maybe another tool gets involved.

Every step can add more context and more inference. From the developer’s perspective, the agent is still working on one task. From the infrastructure perspective, that one task may have turned into dozens of model interactions.

And somewhere in the background, the cost is still increasing.

That is where the architecture starts to matter.

The Expensive Part Is Often Everything Around the Model

There is a tendency to treat AI cost as something dictated by the model provider. Some of it is. A surprising amount of it is dictated by the application.

How much context are you sending? How often are you sending it? How many times can an agent retry? When does a run stop? Does every task need the most capable model available?

Can failed work resume, or does the agent have to start over? Are deterministic tasks being handled with code, or are we paying a model to reason through them every time?

Those are engineering decisions.

And when you multiply them across hundreds of engineers, thousands of tasks, and millions of model calls, small architectural decisions can create very large costs.

This is why we would be careful about concluding that AI itself has hit a productivity ceiling. Jellyfish found enormous differences in token efficiency between developers and workflows. If the model were the only meaningful constraint, we would expect those economics to look much more consistent.

They do not.

The same underlying models can produce wildly different cost profiles depending on how they are used. That suggests there is still a lot of engineering work left to do between the model and the outcome.

Stop Measuring Consumption as Productivity

This also exposes a problem with the way some organizations have measured AI adoption.

Token consumption is easy to measure. Engineering productivity is not. So token usage became a convenient proxy.

But tokens are an input.

Imagine evaluating a Kubernetes workload based on how much CPU it consumed. Or deciding which database was performing best based on which one generated the largest cloud cost. Nobody would call that productivity.

Yet with AI, we have sometimes treated consumption itself as evidence that something useful is happening.

A better question is what each unit of consumption produced. For a development team, that might mean looking at cost per accepted pull request. For another workflow, it could be cost per resolved incident, completed migration, generated test suite, or successfully completed task.

The exact metric will change. The principle should not.

Measure what came out, not simply what went in. Industry coverage has started framing this shift as moving from tokenmaxxing to valuemaxxing — the same idea Fast Company summarized as putting outcomes ahead of raw token volume.

What We Would Look at First

If an engineering organization came to us tomorrow and said, “Our AI costs are growing rapidly and we cannot explain why,” we would not start by telling them to use less AI. We would start tracing where the money is going.

Look at context first. If every interaction sends enormous amounts of repository context back to a model, you are paying for that decision repeatedly. Give the model what it needs to complete the task, not everything it might possibly need.

Then look at loops. Every autonomous process needs boundaries. Agents should have limits on retries, token consumption, tool calls, or some combination of the three. “Keep trying until it works” is not much of a production strategy.

Then look at model selection. Not every task needs your most capable and expensive model. Some tasks require deep reasoning. Others require classification, summarization, extraction, code transformation, or another predictable operation that a smaller model can handle. And some tasks do not need a model at all.

Finally, look at failure. If an agent gets 90% through a long workflow and fails, does it resume from where it stopped? Or do you pay for the first 90% again?

These sound like AI specific questions. They are not. They are the same questions we have been asking about production software for decades.

Where are the limits? Where is the backpressure? What happens when something fails? What resources does this operation actually require? What does this cost us when we run it ten thousand times?

AI did not make those questions obsolete. It made them expensive to ignore.

This Is Also Where Local Inference Gets Interesting

There is another question worth asking:

Does every inference request need to leave your infrastructure and become a metered API cost?

Increasingly, the answer is no. Open source models have become capable enough that many routine workloads can run locally on hardware an organization already controls.

That does not mean every workload should. Frontier models are extremely valuable when a task actually requires frontier level reasoning. The opportunity is to stop treating every task as though it does.

This is part of the thinking behind Kronk AI, the open source project we have been building at Ardan Labs.

Kronk provides a Go SDK for hardware accelerated local inference and a production ready model server, allowing teams to run open source models directly on their own hardware.

The larger idea is more important than the tool itself:

Inference should be an architectural decision.

Maybe a difficult reasoning task belongs on a frontier model. Maybe a repetitive internal workflow belongs on a smaller local model. Maybe deterministic work belongs in ordinary Go code. Maybe sensitive data should never leave your infrastructure in the first place.

The goal is not “run everything locally.” The goal is to stop assuming that every problem requires the same model, infrastructure, and cost structure.

When routine, high volume work can run locally, the economics change. Instead of paying a marginal token cost every time that workflow executes, you can use compute you already control and reserve expensive hosted inference for the work that actually benefits from it.

That is one reason we think local inference will become increasingly important as engineering organizations move beyond AI experimentation and start operating these systems at scale.

Kronk Does Not Make Bad Architecture Cheap

There is an important caveat. Moving an inefficient workflow onto local infrastructure does not suddenly make it efficient.

An agent stuck in an unnecessary loop is still wasting resources. A massive context window is still massive. Poorly scoped tasks are still poorly scoped.

Local inference changes the economics of where that computation happens. It does not remove the responsibility to engineer the system properly.

That is why we see Kronk as one piece of a larger architecture. Bound the loops. Control the context. Route tasks intelligently. Make long running work recoverable. Measure cost against outcomes.

Then decide where each piece of inference should run. Some of it may belong in the cloud. Some of it may belong on infrastructure you control. The important part is that you made the decision intentionally.

The Next Phase of AI Is an Engineering Problem

The first phase of AI adoption was about access. Give developers the tools. Let them experiment. Find out what becomes possible.

We needed that phase. But the next phase looks different.

Now the costs are becoming harder to ignore. Engineering leaders are being asked what those costs are actually producing. And “our developers are using more AI” is not going to be a satisfying answer for very long. Analysts have already started warning that uncontrolled agent spend can outrun payroll itself — Gartner’s projection, covered by The Register, that AI coding agents could cost more than the developers using them.

That does not mean AI failed. It means AI is becoming infrastructure.

And infrastructure eventually has to answer the same questions everything else in production does: How reliable is it? How observable is it? How much does it cost? Where does it run? What happens when it fails?

And most importantly:

What value does it actually produce?

If your token consumption doubled and your engineering output barely moved, the answer probably is not to buy twice as many tokens.

It is to understand where those tokens went. That is where the engineering starts.

Make Inference an Architectural Decision

If your AI spend is climbing faster than your output, the next move is not more tokens. It is better architecture: bounded loops, controlled context, intelligent routing, and a clear decision about where each inference request should run.

That is the problem space Kronk is built for — a Go SDK and model server for hardware accelerated local inference, so routine workloads can run on infrastructure you control.

Explore Kronk AI

Sources

• Jellyfish, “Is tokenmaxxing cost effective? New data from Jellyfish explains”
• Jellyfish, “AI Token Usage Monitoring”
• TechCrunch, "‘Tokenmaxxing’ is making developers less productive than they think" (April 17, 2026)
• Computer Weekly, “Gartner: AI coding agents will cost more than real developers”
• The Register, “AI coding agents could soon cost more than the developers using them” (June 24, 2026)
• ITPro, “From tokenmaxxing to valuemaxxing”
• Fast Company, “Tokenmaxxing is out, valuemaxxing is in”
• WorkOS, “Nicholas Arcolano on why 10x the tokens buys only 2x the output”
• Kronk
• kronk-cli, GitHub