The AI bill has arrived: is limiting usage the answer?

Routing, throughput, and governance

The cost of artificial intelligence is not an expense that can be reduced by decree. It is an architectural decision and, when it involves sensitive data and operations that must be auditable, also a governance decision. Selecting the right model for each task and using self-hosted, open-source models are two strategic levers.

The bill has arrived#

Companies that spent two years encouraging widespread use of generative AI have started applying the brakes. A New York Times report [1], written by Eli Tan and republished by InfoMoney in June 2026, shows Amazon, Walmart, Uber, Cisco, and Meta adopting mechanisms to contain spending that grew faster than expected. In the first four months of 2026, Uber spent the budget it had set aside for the entire year and began imposing monthly limits on AI programming tools. Meta limited usage after an “exponential” cost increase and shut down internal “tokenmaxxing” rankings, a competition generating artificial consumption. Amazon took down its usage dashboard and began directing people toward an internal tool. The message everywhere is the same: use less.

But is that the right answer? Rationing treats the symptom and ignores the cause.

AI cost is not a cash leak that can be stopped by decree. It is a consequence of architectural decisions that, often, no one made deliberately. And when sensitive data is involved, it is a governance issue before it is a financial one.

Two levers produce better results than any per-employee spending cap and still receive little attention: defining which model executes each task and deciding where each model will run.

The trigger is technical before it is financial#

The shift did not start in finance, but when companies replaced chatbots with agents.

While an assistant merely answers, an agent plans, invokes tools, evaluates the result, and repeats the cycle until the task is complete. Each step in that loop consumes tokens, making an agentic interaction significantly more expensive than a conventional chatbot answer. According to the Goldman Sachs report Decoding the Agentic Economy [2], as reported, an always-on agent may consume more than 100,000 tokens a day, compared with about 1,000 tokens for a chatbot. Per task, consumption may be approximately four times higher, and in multi-agent architectures that cost multiplies again.

At the same time, billing shifted from a fixed subscription to tokens. Every automated flow became a visible line on the invoice, and chief financial officers and boards began seeing the spending for the first time. The market had been taught to treat AI as cheap or free, but per-token billing showed that it is not.

Hence the problem with a per-person cap. Limiting employees treats a strategic capability as a cash leak: it cuts cost and, alongside it, value. In any organization where AI already routes customer service, supports risk classification, or underpins products, it is the crudest instrument available, and in a regulated environment where AI operates within auditable processes, it is also the riskiest. The right question is not how much each employee may spend, but which model executes which task, and where each one runs.

First lever: model routing#

The greatest waste in generative AI is usually not volume, but running the most expensive model on every task.

The major AI labs organize their models into tiers. At the top, frontier models (Claude Opus, GPT-5.5, and Gemini 3.1 Pro) deliver greater reasoning capability under ambiguity, planning, and execution of complex tasks. In the middle tier, models such as Claude Sonnet, GPT-5.4 mini, and Gemini 3.5 Flash offer a better balance of capability, latency, and cost for most operations. At the bottom, Claude Haiku, GPT-5.4 nano, and Gemini 3.1 Flash-Lite serve narrow, high-volume tasks: classifying, extracting, routing, and answering trivial questions. The price difference makes the choice consequential. In Anthropic's current pricing table [3], Opus costs US$25 per million output tokens, versus US$5 for Haiku, about five times as much. The gap reaches an order of magnitude (ten times or more) when comparing the top of the market with economical open models such as DeepSeek and Qwen. Across either comparison, the conclusion is the same: paying frontier prices for trivial work is pure waste.

The discipline separating lean operations from an inflated bill has a name: routing. It consists of deciding, task by task, which tier each deserves. An intent classifier does not need a frontier model; an analysis of ambiguous legal text might. When everything defaults to the most capable model, the organization pays reasoning prices for work that a model five times cheaper would perform just as well.

How a router is built in practice#

A comprehensive literature review on dynamic routing and model cascading [4] organizes the problem along three axes:

When the decision is made: before generation, after an initial answer, or in a multistage process, such as cascades that begin with a cheaper model and escalate to a more capable one when the initial answer fails to meet the defined criteria.
Which information is used: query characteristics, model metadata, signals extracted from the answer, and accumulated feedback.
How the decision is calculated: using heuristics and fixed rules, supervised classifiers, bandit algorithms, or reinforcement learning.

The most common concrete pattern is simple. A cheap classifier reads the query and estimates complexity or intent; based on the result, it dispatches to the economical, balanced, or frontier tier; and a fallback covers what slips through. vLLM Semantic Router [5] does this with a ModernBERT classifier, sending queries requiring reasoning to chain-of-thought models and trivial ones to standard inference. Reference implementations exist: LMSYS's RouteLLM [6] is the canonical open-source example; OpenAI's GPT-5 carries an internal router between tiers; Azure sells model routing as a service.

I have already detailed three router architectures in “Three routers, three different problems,” discussing DSPy, a custom semantic router, and a ready-made library. There, routing is about intent within a single assistant: deciding whether a message is a catalog search or support request. Here, routing is about models, based on cost and capability. The object of the decision changes, but the mechanics of a cheap classifier as the entry point remain the same.

The classifier is the single point every request passes through; that also makes it the natural place for recording and policy (revisited below as a control point).

Well-designed routing does not reduce what AI delivers: it reduces what it charges to deliver the same result.

Second lever: self-hosted open source#

The next layer of savings replaces provider consumption with execution on your own infrastructure. Self-hosted open-source models, such as Llama, Mistral, and increasingly the Chinese DeepSeek and Qwen, run inside the company's environment without the provider charging per token.

Adoption of these models is no longer marginal, and the size of the leap deserves precision. Data from OpenRouter [7], a platform routing calls for thousands of developers, shows Chinese open-source models moving from about 1% at the end of 2024 to the largest share among the most-used open models. In some 2026 slices, looking at volume among the ten most-consumed models, they reach 46% to 61% of the identified total. This does not mean they already account for most global consumption: proprietary models still dominate the aggregate. Among open-source models, however, leadership has shifted, driven by aggressive pricing, efficient architectures, and the flexibility to run self-hosted models with access to their weights.

The switch is not magic. By adopting self-hosted models, a company replaces per-token cost with infrastructure, engineering, and operational costs: GPUs, MLOps, security, observability, and teams capable of responding when something fails — without an external provider's SLA to fall back on. And one detail determines whether the economics really work: self-hosted model economics are, in essence, a throughput equation.

The economics of self-hosting are a throughput equation#

The unit cost of a token in a self-hosted operation is, in simplified terms, the total infrastructure cost divided by the volume of tokens it can serve over the same period:

cost per token = total infrastructure cost ÷ total tokens served

The more simultaneous requests a GPU can process while maintaining acceptable latency and quality, the larger the denominator and the lower the cost of each token. Essentially, all inference-server engineering (batching, quantization, parallelism, memory management, and attention-cache optimization) exists to raise that throughput.

This makes break-even calculable. Analyses published in 2026 on self-hosting versus APIs [8] estimate the break-even point relative to premium APIs, in the Claude Sonnet or GPT-4o class, at between 5 and 10 million tokens per month. Compared with lower-cost APIs, such as DeepSeek and mini models, break-even rises to between 50 and 100 million monthly tokens: in these cases, API tokens are already too cheap for dedicated infrastructure to be easily justified.

To size the upper bound, an A100 GPU serving a model with approximately 70 billion parameters using vLLM can reach between 1,000 and 3,000 tokens per second, depending on the model, quantization, context size, batching, and workload profile. At a sustained average of 1,500 tokens per second, theoretical capacity reaches approximately 3.9 billion tokens per month.

But the actual cost is not limited to the GPU. Acceleration hardware may represent only 30% to 40% of total cost of ownership. The rest involves networking, storage, redundancy, power, security, observability, MLOps, and operations. A realistic estimate may therefore require a multiplier of 2.5 to 3 times the hardware-only cost, plus 10 to 20 engineering hours per month.

The conclusion is direct: below break-even, maintaining self-hosted models is usually justified by data control, residency, security, or regulatory requirements, not necessarily price. Above it, with consistent infrastructure utilization, cost per token can fall quickly.

The engine that determines the curve's slope#

Open-weight models do not enter production by themselves. Between the model and application, an inference engine is needed to manage memory, concurrency, and token generation. vLLM has become the de facto open-source standard for this layer. Two of its mechanisms help explain how it increases the cost equation's denominator (often by two to four times).

The first is PagedAttention. One of the main memory bottlenecks in LLM inference is the KV cache, the attention cache that grows with every token processed or generated for each active sequence. Previous systems reserved contiguous memory blocks based on each sequence's expected maximum size. Because few requests used the entire reserved capacity, much of the memory remained idle or was lost to fragmentation. The original vLLM paper [9] notes that this allocation model could waste between 60% and 80% of reserved memory. PagedAttention reorganizes the KV cache into noncontiguous blocks, similarly to an operating system's virtual-memory paging. Waste can thereby fall to about 4% [10]: with 16-token blocks, loss is practically limited to the final partially occupied block of each sequence. In practice, memory that previously remained reserved and idle can accommodate more simultaneous requests on the same GPU. The result is greater throughput and a lower cost per token served.

The second mechanism is continuous batching. In a static batch, new requests enter only when every sequence in the group finishes. Because answers may have different lengths, the GPU ends up waiting for the slowest sequences while some capacity remains idle. Continuous batching operates at each generation step: on every iteration, completed sequences leave the batch and are replaced by new requests, keeping the GPU busy and making better use of available capacity.

In the static batch, B finishes early and stays idle until C, the slowest, ends; in continuous batching, each free slot receives a new sequence at the next generation step.

In the original evaluation, vLLM's combined optimizations increased throughput by two to four times compared with inference systems then considered state of the art, while maintaining similar latency. The same hardware can serve more traffic, and by the throughput equation, average cost per token decreases in the same direction. The actual gain, however, depends on the model, prompt and answer lengths, concurrency, and latency target.

vLLM also reduces the friction of adopting self-hosted open-source models. It provides a server with APIs compatible with the OpenAI standard, distributes large models across multiple GPUs through tensor parallelism, and supports different quantization techniques, such as FP8, INT8, AWQ, and GPTQ, to reduce memory consumption and increase throughput.

In a simple integration, the same client used to access the OpenAI API can be pointed to a local server:

The code that already called the closed API stays the same; the endpoint changes, not the application.

vLLM is not alone. NVIDIA TensorRT-LLM is an alternative oriented toward extracting performance from NVIDIA GPUs, with extensive optimization capabilities but greater dependence on the manufacturer's ecosystem and hardware. SGLang stands out for workloads with shared prefixes or contexts across requests: its RadixAttention identifies common segments and reuses the corresponding KV cache, avoiding repeated processing. Hugging Face TGI, for years a natural option in this comparison, entered maintenance mode in March 2026. Hugging Face itself began recommending engines such as vLLM, SGLang, llama.cpp, and MLX for new implementations.

From an economic perspective, the consequence is direct: the inference server stops being an implementation detail. It determines how many tokens each GPU can deliver within the latency target and, therefore, whether a self-hosted architecture makes economic sense.

The trade-off is multidimensional#

Comparing alternatives solely on price per token is the mistake that produces both unexpected invoices and premature migration to dedicated infrastructure. The decision involves multiple dimensions and, when there is sensitive data, regulatory requirements, or a need for auditing, several carry more weight than unit cost.

The table's assessments are qualitative. The weight of each dimension varies with the workload, volume, criticality, and compliance requirements.

Frontier models maximize capability; economical models reduce cost and latency; self-hosted open models increase control and reduce lock-in, but demand greater operational effort.

No alternative wins on every axis. Frontier models offer greater capability for complex tasks and require little operational effort, but have higher unit cost and less direct control over the processing environment. Economical models balance cost, speed, and simplicity, although they may not sustain tasks requiring deeper reasoning or greater reliability. Self-hosted open-source models offer greater control over data, infrastructure, and operations. In return, they transfer full responsibility for capacity, availability, security, guardrails, observability, and platform evolution to the organization.

The right architecture rarely consists of just one option. It combines tiers and assigns each workload to the most economical model that still meets its capability, quality, latency, security, and compliance requirements.

Where cost becomes a risk decision#

This is where the conversation changes departments. The choice of model, execution location, and routing layer determines three things simultaneously: where data travels, which evidence remains available for auditing, and who takes responsibility for guardrails. In regulated sectors such as financial services, healthcare, government, and law, these variables are matters of governance, security, and operational risk before finance. What may be merely good practice in e-commerce can become a necessary control in these environments, depending on process criticality and the sensitivity of the data involved.

Start with data flow, not just the self-hosted label. Self-hosted inference may keep data within the organization's perimeter, but only when the entire chain remains under its control: inference server, embeddings, logs, telemetry, guardrails, storage, and other integrations. If any layer sends information to an external service, part of the flow remains outside the perimeter. The LGPD [11] does not mandate local model execution or prohibit cloud use; it requires a legal basis for processing and appropriate technical and administrative measures to protect personal data. In the financial sector, CMN Resolution 4.893/2021, as amended by CMN Resolution 5.274/2025 [12], establishes cybersecurity and governance requirements for contracting data processing, storage, and cloud-computing services. Neither regulation expressly mentions AI inference or mandates self-hosted models. Keeping certain data and workloads within the perimeter is an architectural decision that may reduce third-party exposure and facilitate compliance with security, control, and auditability requirements, not a literal regulatory order.

There is, however, a convergence point the market still underuses: the same routing layer deciding which model receives each task can serve as a central policy-enforcement point. Combined with a gateway or unified inference endpoint, every request passes through a layer where it is possible to authenticate the user, classify data, mask personal information, enforce access rules, select the model, and record the path taken. This trail does not emerge automatically. It must be designed: structured logs, request correlation, policy versioning, recording the selected model, decision justification, metrics, and appropriate evidence retention. Without that instrumentation, centralizing traffic merely creates a bottleneck; with it, a verifiable control point is created.

It is no coincidence that semantic routing is moving closer to the inference ecosystem itself. vLLM Semantic Router already combines model selection, PII detection, protection against malicious prompts, semantic caching, metrics, and tracing. A component created to optimize cost and performance is also beginning to assume security, observability, and governance functions. The same principle underpins the output guardrail I described in “Regulatory guardrails in an investment assistant”: establishing explicit points where policy can intervene. In that case, control is at the output, in the critique_node evaluating the answer before it reaches the customer. Here, it is at the input, in routing and the inference gateway.

These are different points in the same pipeline, guided by the same logic: relevant decisions cannot remain scattered, implicit, or impossible to reconstruct. They must be centralized when necessary, versioned, tested, and recorded.

Under this framing, cost optimization stops being merely expense reduction. It becomes a governance capability: knowing which model processed each request, under which policy, over which data, and under whose responsibility. The regulator does not require vLLM or semantic routing, but expects the organization to demonstrate its controls, decisions, and management of the risks involved.

How to decide#

The operational question should not be how much the organization may spend, but where each workload should run. A few criteria resolve most cases: how sensitive the data is and whether it may leave the perimeter; the task's actual complexity and whether it needs a frontier model or is well specified; the volume and whether it is steady or irregular; latency tolerance; and the required auditability level. The answers point almost naturally toward a model tier and hosting mode.

The result of this exercise is not using less AI. It is building a portfolio in which each flow runs on the most economical option that still meets capability, quality, latency, security, and compliance requirements.

An organization doing this does not need to begin with a per-employee spending cap. The cost limit becomes a consequence of an architecture that already knows what is worth executing, on which model, and in which environment, not a premise imposed from the top down.

What separates rationing from architecting

The report opening this text describes a restriction policy. It contains the loss, but also limits the gain. Organizations leading the next phase will not treat AI as a resource to ration. They will architect it, turning an opaque invoice into a governed, measurable portfolio capable of standing up to audit.

Time works against those who choose only restriction. According to the Decoding the Agentic Economy report from Goldman Sachs, as reported, global token consumption could grow 24-fold by 2030, driven by advances in agents. Facing expansion of that magnitude, how long can a per-employee cap contain the bill without also containing adoption?

Rationing limits the symptom. Architecting addresses the cause.

This article presents a technical and architectural analysis and does not constitute legal, regulatory, or compliance advice or an opinion. Interpretation and application of the LGPD, Central Bank regulation, and other frameworks applicable to each sector should be carried out jointly with the organization's legal, compliance, privacy, and information-security departments.

Sources

InfoMoney - “The bill has arrived: why technology companies are now slowing AI usage”, https://www.infomoney.com.br/business/global/a-conta-chegou-por-que-as-empresas-de-tecnologia-agora-freiam-uso-de-ia/
Goldman Sachs - “AI Agents Forecast to Boost Tech Cash Flow as Usage Soars”, https://www.goldmansachs.com/insights/articles/ai-agents-forecast-to-boost-tech-cash-flow-as-usage-soars
Anthropic - Claude pricing, https://platform.claude.com/docs/en/about-claude/pricing
Dynamic Model Routing and Cascading: A Survey, https://arxiv.org/html/2603.04445v2
vLLM Semantic Router, https://vllm-semantic-router.com/
RouteLLM: Learning to Route LLMs with Preference Data - LMSYS. https://arxiv.org/abs/2406.18665
OpenRouter - State of AI. https://openrouter.ai/state-of-ai
TokenMix - Self-Host LLM vs. API: Break-Even Analysis (2026), https://tokenmix.ai/blog/self-host-llm-vs-api
Kwon et al. - Efficient Memory Management for Large Language Model Serving with PagedAttention, SOSP 2023, https://arxiv.org/abs/2309.06180
Runpod - vLLM PagedAttention and Continuous Batching, https://www.runpod.io/articles/guides/vllm-pagedattention-continuous-batching
Law 13.709/2018 - Brazilian General Personal Data Protection Law https://www.planalto.gov.br/ccivil_03/_ato2015-2018/2018/lei/L13709compilado.htm
Central Bank of Brazil - CMN Resolution 4.893, February 26, 2021 https://www.bcb.gov.br/estabilidadefinanceira/exibenormativo?numero=4893&tipo=Resolu%C3%A7%C3%A3o+CMN
Central Bank of Brazil - CMN Resolution 5.274, December 18, 2025, https://www.bcb.gov.br/estabilidadefinanceira/exibenormativo?numero=5274&tipo=Resolu%C3%A7%C3%A3o+CMN
vLLM - official documentation, https://docs.vllm.ai/
Hugging Face - Text Generation Inference, https://github.com/huggingface/text-generation-inference

Cited articles#

“Three routers, three different problems: DSPy, Semantic Router, and RouteLLM” https://www.linkedin.com/pulse/tr%25C3%25AAs-routers-problemas-diferentes-dspy-semantic-router-bruno-vicco-dbi7f/
“Regulatory guardrails in investment assistants: CVM, Anbima, and AI” https://www.linkedin.com/pulse/guardrails-regulat%25C3%25B3rios-em-assistentes-de-cvm-anbima-e-bruno-vicco-rksff/