Skip to main content

Cost per request: 26% of traffic is an AI decision

November 16, 20265 min. read
Cost per request: 26% of traffic is an AI decision

A quarter of the traffic through a production gateway is already decision, not text generation: in October 2026, 26% of the requests went to decision models, which accounted for 0.53% of spend. The data moves the question. The AI cost per request, not the price per token, is what decides the invoice.

How was the index measured, and where does it stop?

The index measures anonymized, aggregated traffic from a gateway, above 100 trillion tokens per month, with the series closed in October 2026. Spend is estimated at list prices. It is not the AI market, and it is not the invoice of whoever reads this text.

The data comes from the November 2026 production index, published by Vercel on November 9. Two limitations matter. The perimeter is one gateway, not the market. And the money is an estimate at list prices, because negotiated discount, annual contract and commitment credit do not enter the public price table the source uses to estimate spend.

Neither limitation invalidates the reading.

Table 1. What the index measures and where it stops

WhatUnitLimit
Requests by model class% of total requestsOnly the traffic that passes through this gateway
Spend by class% of total spendEstimated at list prices, not an invoice paid
PeriodMonthly series, closed in October 2026Published on November 9, 2026
BaseAbove 100 trillion tokens per monthAggregated and anonymized, with no view of the reader's load

A mix portrait does not need to be an invoice to say which share of the volume and which share of the money each model class occupies. That proportion is what survives the fact that the books belong to a vendor.

What the index shows: 26% of requests and 0.53% of spend

Decision models went from zero in August to 26% of all requests in October, and accounted for 0.53% of spend. In the same move, language model requests fell by more than a fifth. A new class now carries a quarter of the volume.

A decision model does not write text: it classifies, scores or chooses between options. The class entered the gateway in mid-September and reached October with 15 models from 8 vendors.

The three numbers of the month:

  1. Decision models accounted for 26% of requests and 0.53% of spend, against zero in August.
  2. Language model requests fell by more than a fifth in the same move.
  3. 11% of paying teams used a decision model without sending a single request to a text-generating model.

That is the strange number of the series. There is AI operation in production that generates no text at all, and almost a third of paying teams already used a decision model in production in October.

Table 2. Requests against spend by class, October 2026

ClassShare of requestsShare of spend
Decision models26%0.53%
Language models, embeddings, reranking and media74% (derived by subtraction)99.47% (derived by subtraction)
inline-01.png

The chart is the whole piece: a quarter of the requests and half a percent of the money. Whoever looks only at spend concludes that decision is irrelevant; whoever looks only at requests concludes the opposite.

How much does a decision request cost compared with a generation one?

The arithmetic closes an order of magnitude: a decision request costs about 66 to 70 times less than a language model request that does the heavy generation. The source publishes that this cost is about one seventieth. The two readings, one calculated here and one published, point to the same place.

The arithmetic uses 100 requests and 100 units of spend. The decision class takes 26 requests and 0.53 of spend: 0.53 divided by 26 gives 0.0204 per request. The remainder takes 74 requests and 99.47: 99.47 divided by 74 gives 1.3442. The ratio between the two is 65.9, or approximately 66.

About 66 to 70 times: the percentages are rounded and the residual carries embeddings, reranking and media. Order of magnitude, not measurement.

The second account is narrower. Almost all of the decision request volume went to Jev, from TypeSafe AI, with almost four times the request volume of Anthropic. For every US$ 1 spent on Jev, teams spent US$ 64 on Anthropic models. Four requests per US$ 1 give US$ 0.25; one per US$ 64 gives US$ 64. With the two approximations of the source, the ratio sits at about 1/256.

Table 3. Relative cost per request between two routes

RouteRequestsSpendCost per request
Jev, from TypeSafe AIAlmost 4 timesUS$ 1About US$ 0.25
Anthropic models1US$ 64About US$ 64
Ratio between the routesAbout 1/256

With more than two orders of magnitude per request, choosing a route by the price table per million tokens measures the variable that weighs least. The model that returns only decisions explains why the class exists; the account above explains why it changes the invoice.

The cost per request that the token price does not include

The inference cost per request has five components: input and output tokens, retry amplification, fallback delta, operation of the routing layer and amortized observability. The price per token table covers the first and ignores the other four, so a table that covers only the first describes a fraction of what the invoice charges.

The second component, retry amplification, is the most ignored one. Each new attempt pays in full, and a request that fails twice before succeeding costs three times what the table promises. Cost per attempt turns retry into money, and the record has to prove the attempts of each request.

The third is the fallback delta: the difference between standby and primary, applied to the fraction of requests that migrate. The standby is rarely the cheapest, because it exists for the day the primary does not answer. Model fallback criteria say when the migration happens; the account says what it costs.

The fourth is the LLM gateway overhead: normalization, intent classification, model selection, distribution and failover, with a fixed part and a part that follows the volume. The fifth is observability: traces, evaluations and storage, divided by the requests of the period. Amortized, it looks small; not amortized, it becomes a platform line nobody associates with inference.

The formula:

cost per request = (t_in × p_in + t_out × p_out) × r + f × delta + c + o

t_in and t_out are input and output tokens, p_in and p_out the prices per token of each route, r the average number of attempts, f the fraction that falls on the standby, delta the cost difference between standby and primary, c the cost of the layer and o the amortized observability. No term has a universal value.

Table 4. The five components of cost per request

ComponentWhat entersWho controls it
Input and output tokensVolume consumed per route and published priceModel vendor
Retry amplificationEach new attempt pays in fullApplication engineering
Fallback deltaDifference between standby and primary, times the migrated fractionAvailability policy
Routing layerNormalization, classification, selection and failoverPlatform operation
ObservabilityTraces, evaluations and storage, amortizedPlatform operation
inline-02.png

This article converts traffic mix into an account per request; latency, memory and the overhead of the layer are in the operational cost of an AI gateway.

Why the adoption cohort changes the number

Adoption rises with volume: a little more than one in five teams below 10 million tokens per month uses a decision model, almost half between 100 million and 1 billion, and seven in ten above 10 billion. Below the first step, the layer tends to be a new fixed cost.

The ladder describes a threshold. Whoever runs 10 million tokens per month has the volume to justify a classifier and does not have the volume to dilute the cost of keeping the layer. The account changes sign as the denominator grows.

Table 5. Decision model use by volume cohort

Token cohort per monthTeams that used a decision model
Below 10 millionA little more than one in five
Between 100 million and 1 billionAlmost half
Above 10 billionSeven in ten

Model concentration changes with size. The 1% that spends the most uses a median of 29 different models. The tenth that spends the least uses one.

There is a reading of the index that runs against the conclusion it sells. The gateway vendor sells routing, and the easy reading is that routing always pays. The ladder contradicts it: at the base, most teams do not use a decision model, and the probable explanation is the arithmetic of a small single-model load, where the new layer costs more than it returns.

The six-week half-life changes the question

More than half of the language requests in October went to models less than 60 days old, against fewer than three in ten before September. The source sums the effect in one expression: a half-life of about six weeks. A frozen routing policy, in this regime, is a cost decision by omission.

Turnover has its own numbers. A new version arrives every 61 days in the family median, a typical model reaches its peak in about a month, and four weeks after the peak three in four teams that used it had already migrated. Half of the teams with 10 million tokens or more changed their most used model in October alone, and six in ten went to another lab.

Table 6. Model turnover in October 2026

IndicatorValue
Language requests on models less than 60 days oldMore than half, against fewer than three in ten before September
Interval between versions in the family median61 days
Time to the peak of use of a typical modelAbout one month
Teams that migrated four weeks after the peakThree in four
Teams with 10 million tokens or more that changed their main model in OctoberHalf
Migration to another labSix in ten

The denominator the buyer measures in January is not the same one from March. A price per token frozen in a twelve-month contract ages faster than the model it describes, which is why the test below repeats the measurement in 30 days.

The five-step test before closing the routing architecture

The test uses the load itself as a sample and takes five steps: count requests by class, calculate the full cost, measure retry and failover, compare the accuracy of the classifier on both model types and repeat in 30 days. It fails if the classifier shows no gain or if retry consumes the difference.

Each step has a verifiable output and a failure criterion. Without both, the test turns into a meeting.

  1. Count requests by class over seven days, separating decision, generation, embedding and media, with a trace per call. Without an LLM call trace, the aggregated log hides the class and the retry. Fails if the count does not separate the classes.
  2. Calculate the full cost per request with the five components, using the invoice actually paid, not the table. Fails if the number comes from the pricing page.
  3. Measure the retry rate and the failover rate and convert them into extra requests. Fails if the amplification consumes the difference between the new route and the current one.
  4. Run the classifier on both model types over a labeled sample of the operation itself and compare the accuracy. Fails if it does not beat the rule already in the code.
  5. Repeat in 30 days and record what changed. Fails if the difference between the two measurements has no identifiable cause in the load, because that indicates the measurement did not capture what changed in the operation.

The test does not decide whether routing models by task is good. It decides whether the route the buyer is about to contract is better than the one already in place, with their own load. A failing result is information.

What this changes for buyers in Brazil

For whoever pays in reais, the cost per request is the unit that survives the exchange rate and the import taxes, because both enter the same denominator. The invoice in reais with a tax document is what allows comparing the list-price estimate with the outlay of the month.

A price per token in dollars says nothing about cash until it passes through exchange rate, taxes and payment date. The cost per request, measured on the invoice that arrived, has already passed through all of it. It changes currency. It does not change logic. It is the line the buyer defends in front of finance.

This is where Nexforce Router enters, as a route decision product, not as the source of any number in this article. Router exposes the main model families behind an API compatible with the one the application already consumes, with routing by cost, performance, latency and context, automatic failover, a spend cap per key and a trace of each call. The index is third party and does not measure Router; Router delivers the place where the routing policy stays written, with an owner and with a limit.

Frequently asked questions

Does cost per request replace cost per token?

It does not replace it, because the two accounts answer different questions. Cost per token is the raw material of a route; cost per request is what the invoice charges per request, after retry, fallback and layer. The accounts of 66 to 70 times and of about 1/256 only exist per request.

Does the index apply to my load?

It applies as a reference for mix, not as a forecast. The 26% of requests and the 0.53% of spend describe aggregated traffic from one gateway, estimated at list prices. Applying those percentages to a specific company only makes sense after counting requests by class in its operation.

Does a decision model serve any stage?

It serves stages with a discrete answer, such as classifying, scoring or choosing between options. It does not serve to write the final answer to the customer. The index shows 11% of paying teams using a decision model with no request at all to a text-generating model, which points to entire operations built on decision.

References

Primary source consulted on 2026-11-16:

  1. November 2026 production index (Vercel, 2026-11-09): https://vercel.com/blog/ai-gateway-production-index-november-2026

Further reading, consulted on 2026-11-16:

  1. Operational cost of an AI gateway: https://nexforce.ai/en/blog/measure-ai-gateway-operational-cost
  2. Cost per attempt: https://nexforce.ai/en/blog/ai-inference-cost-per-attempt
  3. TypeSafe launches Jev: https://nexforce.ai/en/blog/typesafe-jev-calibrated-decisions-llm-routing
  4. AI model fallback: https://nexforce.ai/en/blog/ai-model-fallback-acceptance-criteria
  5. LLM call trace: https://nexforce.ai/en/blog/llm-call-trace-audit-request-token

The index is monthly and revises the previous months, so the percentages on this page may change in the next edition.

The account that changes the next architecture meeting

The index does not say that routing pays. It says that a new class accounts for a quarter of the requests with half a percent of the money, and that the unit able to explain that difference is the request, not the token. Whoever walks into the meeting with the account per request in hand discusses cost. Whoever walks in with the price table discusses price, and leaves the room believing they decided.

Nexforce

Accelerate your company'sbusiness and operational efficiency

We design the technology of tomorrow to boost your business operational scale

Talk to a Specialist

Related articles