LLM gateway: why does latency grow with context?

An LLM gateway spent 553 ms of TTFB before calling the model in a 439,945-token conversation, with 334 turns and 18 tools. After cutting three pre-flight steps, the same path answered in 35 ms, in a benchmark published on 3 October 2026 by an open-source LLM gateway project.

The measurement separates two things the invoice merges: the work the gateway runs between the request arriving and the hand-off to the provider, and the time the model takes to emit the first token. With a provider that answered immediately, whatever is left on the stopwatch belongs to the path. In a long coding session, that path charges for the accumulated history, and the history only grows: the most expensive step re-read the whole conversation to answer yes or no about cache. The bottleneck sat before the call.

Which numbers does the 3 October benchmark publish?

Nine numbers, one source, two routes and one machine. Each line can be quoted on its own as long as it carries the attribution: a benchmark published on 3 October 2026 by an open-source LLM gateway project, run in a local environment.

1. 553 ms. Median TTFB of the chat completions path before the fix, with the provider answering immediately.
2. 35 ms. The same path, afterwards.
3. 94%. Reduction in the median TTFB of the chat completions path.
4. 542 ms. Median TTFB on the Messages endpoint, before the fix.
5. 40 ms. The same endpoint, afterwards.
6. 439,945 tokens. Size of the conversation used in the test.
7. 334 turns. Request and response pairs in the same conversation, with tool results in between.
8. 18 tools. Tools available in the session.
9. Three requests. The median at each point, measured after a warm-up.

Two caveats apply. The numbers come from two routes measured on a local machine, not from production traffic: there is no p50, p95 or p99, and the median of three requests is the whole statistic. And the 94% belongs to the median TTFB of the chat completions path, in that design.

How was this benchmark run?

The request path ran in Python, with a single healthy deployment, response cache in Redis, the prompt caching check enabled and a test provider that answered immediately. Each point is the median of three requests after a warm-up, on the same local machine, across two routes.

The instant upstream is what gives the design its meaning. With a real provider at the end of the line, TTFB mixes the gateway time with the provider queue and the model prefill, and there is no way to separate the two from the number alone. With something that answers in the same instant, what remains is the path.

The single deployment works on the same principle: with one healthy route there is no decision to make. The test therefore measures the case where the eligibility question decided nothing. What is missing: distribution, tail and price.

Where did the 518 ms the LLM gateway spent before the call come from?

Three steps in the previous version account for the 518 ms. The largest of them was an optional prompt caching check that counted the whole conversation to answer whether the text reached the minimum size the deployment requires to reuse the prefix, which is defined per model and in this measured gateway sits at 1,024 tokens. The other two: prefix hash and cache pin lookup.

The check had a defensible motive: keeping the request on a deployment with cache saves compute on the model side. The problem was the method. The answer is yes or no, and the gateway read everything.

In a coding session with hundreds of turns and tool results, that count could mean hundreds of milliseconds of Python work before the model call. The gateway read the whole history to decide whether it was worth sending the whole history to a deployment with cache.

The 518 ms is the sum of the three steps the new version skips, and the count accounts for the part that grows with the conversation. The 35 ms that remain are not their residue: on the measured path the three steps are skipped entirely, and what stays on the stopwatch is the rest of the path on the same machine, the same thing Figure 1 labels path residue.

The question cost more than the answer.

It is worth separating two caches that appear together in this conversation. The provider prompt cache stores key and value representations and the hashes of the cached blocks, in memory, with a TTL of minutes or of one hour, and not the prefix as text or at rest. KV cache transfer between models is another technique, an infrastructure one, and not prefix reuse per request. The cache decision in the gateway is something else: the question of where the request should land.

Why does the cost grow with the size of the conversation?

The cost grew because the step walked the text that arrived. The work tracks tokens and turns, and agentic load maximises both: every tool result enters the history, the history comes back whole in the next request, and the session that most needs context is the one that pays most.

That is a reading of the mechanism, not a direct measurement. The step walked the whole conversation, so the cost removed tracks the text counted, with the caveat that the source publishes a point, not a curve. A rate in milliseconds per token does not exist, and a regression of overhead against context would be invention.

A routing decision that examines the text consumes the input it decides on. In a short prompt, the arithmetic is irrelevant. In a 439,945-token conversation, the three pre-flight steps added up to 518 ms, and what the model spends after that is not measured.

When price ties, latency becomes the route variable, and the one that decides there is the model's, measured by whoever serves the inference. The latency in this article is that of the path to the model.

What changes when there is more than one deployment?

The path changes, the cache does not. With a single healthy deployment there is no route to choose, and the new version skips the eligibility check, the prefix hash and the cache pin lookup. With several deployments, the count stops at the first message that reaches the minimum size the deployment requires to reuse the prefix. That minimum is defined per model, and the range runs from 512 to 4,096 tokens; in this measured gateway the cut happens at 1,024 tokens. Affinity and hash stay in use.

The fix for the multi-deployment case separates overhead from functionality. Nobody switched prompt caching off. The question kept existing, with a ceiling: it stops counting when the answer is already known. The cost of the LLM routing decision falls because it no longer walks the whole conversation to answer something the first message resolved.

The third path, the /v1/responses endpoint, already ignored the check and stayed approximately stable. Usage accounting did not change in any branch.

Cache read economics in routing is price arithmetic: what it costs to read a prefix that is already cached. What this benchmark measures is the price of deciding who may read the cache: the cost of the decision, not the saving it buys. With an instant upstream there is no real prefill, so the cache gain stays unobservable in this design.

What changes in an application's latency budget?

What changes is the shape of the sum. TTFB is the gateway pre-flight work, plus the network, plus the provider queue, plus the prefill and the first decode of the model. The pre-flight grows with what the rest of the sum consumes. In gateways that do work proportional to the context before dispatch, like the one measured here, overhead measured with a short prompt underestimates the latency of agentic load. In a gateway that chooses a route from metadata, deployment health and a cost table, the pre-flight does not grow with the conversation, and the underestimation does not appear.

The values come from the 3 October 2026 benchmark, median of three requests after a warm-up on the same machine. The Difference column is a subtraction of the published medians, not a new measurement.

The consequence for whoever architects the system is uncomfortable. A latency budget assembled from short-prompt measurement does not hold for agentic load, measured here in a 439,945-token coding session with 334 turns. The LLM routing decision by latency now needs both terms of the sum.

Reading someone else's number requires the same discipline, and router benchmark metrics separate what the measurement covers from what it only suggests. A TTFB of 35 ms with an instant upstream is not what the user feels. It is the floor of the path.

How to measure your gateway overhead with long context

Seven steps, and each one avoids a specific error. The design starts from the 3 October 2026 benchmark, applied to your traffic: a real long conversation, an immediate upstream, a sample large enough for percentiles, concurrency, a measurement boundary, routes measured separately and a comparison with the check on and off.

1. Capture a real long conversation. Use a session that already happened, with turns and tool results. It avoids measuring with a toy prompt and concluding that gateway overhead is small.
2. Point at an upstream that answers immediately. It avoids mixing model generation with path work.
3. Sample enough for p95 and p99. A median of three requests does not describe the tail. Discard the warm-up and publish percentiles. It avoids hiding a p99 that collapses.
4. Measure each route separately. It avoids an average that hides the route that hurts.
5. Measure under concurrency. The benchmark is serial. Token counting in Python inside an asynchronous gateway is CPU-bound work in the same process: it occupies the event loop and blocks every concurrent request on the same worker. It avoids accepting a serial median while the p99 with dozens of simultaneous sessions collapses.
6. Measure inside the process or on the same host. TTFB measured at the client includes TCP, TLS and RTT, and on a 439,945-token body the round trip and the transmission can exceed the measured effect. It avoids attributing to the pre-flight what is network.
7. Repeat with the check on and off. Only the difference between the runs belongs to the pre-flight. It avoids crediting the improvement to another change.

That design accounts for the shape of the load. The distribution in real traffic still calls for the method for p50, p95 and p99 in measuring the operational overhead of a gateway: one piece measures the distribution, the other measures how gateway overhead changes with the conversation.

What the Nexforce Router delivers at this layer

The Router delivers the three things this design requires, and none of them is a promise of lower latency. The Router chooses a route by cost, performance, latency and context, keeps a response and embeddings cache, and maintains a complete trace of every call with logs, metrics and alerts in the same place.

Latency-based selection speaks to the budget described above: whoever decides a route without knowing how much of TTFB is pre-flight optimises the wrong term of the sum. The trace solves the same problem from another angle, because without a per-call record the overhead paid does not appear on the invoice or in the provider dashboard.

It is worth recording the limit. No Router latency number enters here, because no public measurement supports one. The product delivers the layer where the measurement happens.

Frequently asked questions

What is TTFB in an LLM call? It is the time between the start of the request and the first byte of the response, the TTFB definition in the MDN documentation, which holds the same for a model call. In an LLM call it adds five parts: the gateway pre-flight work, the network, the provider queue, the prefill and the first decode of the model.

What is prompt caching and why does it influence the route? It is the reuse of an already processed prefix, so the provider does not recompute the same stretch on every turn, as the Anthropic prompt caching documentation describes. It influences the route because not every deployment reuses the prefix, and eligibility depends on the prompt reaching that model's minimum in tokens. The documentation does not fix a constant: the range runs from 512 to 4,096 tokens and the value is defined per model, so it is worth checking the model you use, and in this measured gateway the cut happens at 1,024 tokens.

Why can counting tokens before the call be expensive? Because counting requires walking the text, and the text of a long session is the heaviest asset on the path. In a 439,945-token conversation, with 334 turns and 18 tools, the count could mean hundreds of milliseconds of Python before the model call, according to the 3 October 2026 benchmark.

Does a 94% reduction mean my gateway can get 94% faster? No. The number holds for the median TTFB of the chat completions path, in a 439,945-token conversation, with a healthy deployment and a provider that answered immediately, median of three requests on a local machine. It measures the removal of three pre-flight steps in that design, which does not include provider queue, network or real traffic.

References and Further Reading

The numeric source is a benchmark published on 3 October 2026 by an open-source LLM gateway project, in a local environment, read in full on 22 October 2026. The attribution is by category and date because the project is a competitor and is not named on any published surface.

• Time to First Byte definition, MDN documentation
• Prompt caching, Anthropic documentation

The internal further reading sits in the body, in the five texts cited across the sections. No link points to the measured project.

The cost that does not make it into the latency spreadsheet

The most expensive item in the latency budget was a question the gateway asked itself. The model call, the provider queue and the size of the response stay out. What weighs is the internal accounting that decides where the request should land, executed on every request, over the text that arrived.

Two actions, in this order. First, measure TTFB with a real long conversation and an instant upstream, because the number measured with a short prompt does not describe the load that matters. Then review what runs before the call and ask, item by item, whether that has to run on every request.

The honest limit: no public data says how often this pattern appears in other gateways. What exists is a dated measurement, on one path, with an explicit design, and a mechanism that any gateway that decides a route can reproduce.

How to cite: a benchmark published on 3 October 2026 by an open-source LLM gateway project, run in a local environment, read on 22 October 2026. Update: a revision when there is a new public measurement of the same path.