KV Cache Transfer Between Models in AI Agents: Cost and Latency Cuts of up to 25x

Cross-model KV cache transfer cuts agent latency by up to 25x

Two papers on arXiv describe methods that move the Key-Value cache directly between different AI models. Nvidia published on August 4, 2026 a closed-form linear mapping that eliminates prefill reprocessing when an agentic workflow switches models, cutting transition latency by up to 25x, with 25.1x measured on the transition from Qwen3 14B to Qwen3 32B at a 32,768-token context. The Cache-to-Cache framework, from Fu et al., published at ICLR 2026, attacks the same bottleneck through neural fusion of tensors.

The computational barrier of autonomous systems is moving to a new level. Until now, any architecture that combined specialized models faced an expensive dilemma. If a lightweight model triaged documents and handed the task to a larger model, the system had to recalculate the entire attention from scratch.

The wasted time was enormous. In long code debugging flows or corporate repository navigation, dense model prefill consumed seconds of GPU time. The Nvidia paper demonstrates that attention computed by a smaller neural network can be converted directly into tensors usable by a larger one.

The advance resolves the central contradiction of contemporary multi-agent architectures. Engineering teams wanted smaller models to save operating resources, but paid such a high time penalty on the handoff that the financial gain disappeared. With mathematical projection of key and value tensors, that penalty falls sharply.

The hidden bottleneck: the handoff rate in multi-model agents

AI agents operate by splitting tasks between smaller triage models and larger reasoning models, but that division runs into the handoff rate. When context transfers, the receiving model must recalculate the entire attention matrix in prefill, discarding the previous GPU work and producing severe delays.

To understand the root of the problem, separate the two structural phases of Transformer inference: prefill and decode. The prefill phase processes input tokens in parallel to build the key (Key) and value (Value) matrices in video memory. That stage consumes a high rate of floating-point operations and is strictly limited by the computational capacity of the chips.

The decode phase emits tokens autoregressively, one at a time. Each new token consults the KV cache already built, which makes this stage limited by the GPU memory bandwidth. When a session reaches tens of thousands of tokens, the accumulated KV cache stores the complete map of contextual relationships.

The agent moves the work. Conventional infrastructure throws that memory map away. The receiving model gets the history as raw text and recalculates every linear attention projection. At contexts of tens of thousands of tokens, that rework represents billions of redundant mathematical operations that raise energy costs and delay the final answer to the corporate user.

The traditional alternative had the first model write a text summary before the handoff. That text route creates another serious problem: it forces a slow decode phase and compresses rich continuous representations into discrete words. Subtle probability and context information simply evaporates along the way.

The Nvidia architecture: ridge regression projection and tensor alignment

The Nvidia research team built a mathematical bridge that converts the KV cache between models of different sizes without rerunning prefill. The method combines source layer selection, inversion of Rotary Position Embeddings, and per-attention-head ridge regression. At a 32,768-token context, conversion between Qwen3 14B and Qwen3 32B took 278 milliseconds, against 6,975 milliseconds for conventional prefill, measured on a node with 8 H100 GPUs at bf16 precision.

The engineering challenge looked nearly insurmountable because of deep structural disparities between models. Networks of different sizes carry different hidden dimensions, variable numbers of attention heads, and their own positional schemes. Nvidia closed those gaps with a process of three well-defined analytical components.

Selecting the most predictive source layers is the first component. A 14-billion-parameter model has no direct one-to-one correspondence with a 32-billion-parameter network. The researchers used the head-averaged coefficient of determination (R²) of a linear regression to identify, for each target-model layer, the most predictive source layers.

The second component removes the positional rotation from the cache. The Rotary Position Embedding (RoPE) encodes the temporal order of tokens by applying two-dimensional orthogonal matrices over pairs of vector coordinates. To let tensors from different models talk to each other, the system mathematically inverts those rotations, recovering activation vectors in a positionally neutral state.

The linear projection itself is the third component: a ridge regression, also known as Tikhonov L2 regularization, applied per attention head. A pretrained weight matrix projects the key and value vectors across the two dimensions. The regularization stabilizes the inversion of the problem's covariance matrix (XᵀX), which becomes nearly singular because the selected source layers are correlated with each other. At the end, the system reapplies the RoPE rotations adjusted to the receiving model's dimensional frequencies and position indices. The whole operation reduces to simple matrix multiplications.

The reach has a stated limit in the paper itself. The technique works within a single model family, on pairs with compatible cache formats, and of the six pairs tested two degrade heavily, Ministral 3B to 14B and 8B to 14B, with 41% to 44% accuracy retention. The paper itself shows a recovery path: a nonlinear MLP-based mapper, trained with the same calibration data, adds 24 to 37 percentage points of HellaSwag retention on those degraded pairs. Among the four working pairs, retention ranges from 73% to 98%, and the Qwen3 14B to Qwen3 32B transition records 97.6% average accuracy retention of the receiving model across five benchmarks.

Converting tensors through linear matrices consumes a negligible fraction of GPU power compared with processing the full attention.

The Cache-to-Cache framework: attention fusion without the text bottleneck

The Cache-to-Cache framework introduces a neural fusion layer that connects the latent representations of specialist models directly in GPU memory. Instead of forcing agents to exchange plain-text reports, C2C transfers pure attention tensors, reaching an average 2.5x speedup and a gain of up to 5.4 percentage points of accuracy over intermediate text communication.

The gain is twofold.

Serializing intermediate reasoning into running text is a documented information bottleneck in the machine learning literature. When a model summarizes its conclusions in text form, it flattens continuous distributions of certainty into isolated symbols. The next model reads the text, but loses the trace of the uncertainties and latent connections the previous model computed during its initial exploration.

The C2C fuser operates directly on the activation tensors present in video memory. It receives the KV cache blocks generated by specialist agents and fuses their representations through a neural cache fuser with a learnable gating mechanism, which selects the target model's layers that benefit from cache communication. That fusion preserves nuances no text prompt chain can transmit.

The practical results confirm the superiority of this approach over conventional flows. On multi-step problem-solving benchmarks, C2C cut aggregate latency by 2.5x and raised the hit rate by up to 5.4 percentage points over intermediate text communication. The agents cooperate without waiting for intermediate text generation.

Technical comparison: traditional recomputation versus cache transfer

Replacing redundant prefill with direct tensor transfer techniques changes the compute consumption profile of AI agents. Linear methods and memory fusion cut GPU overhead, letting multi-model architectures run with response times compatible with interactive real-time systems.

The table below compares how systems behave under the existing approaches and the new tensor preservation techniques:

The data expose the inefficiency of the conventional model. The math does not close. Paying for prefill twice on every agent-to-agent transition kills interactive corporate products that demand fast answers over long technical documentation or codebases. Direct transfer turns the handoff between models from an expensive obstacle into a nearly instant operation.

Impact on enterprise infrastructure: open models versus proprietary APIs

Direct KV cache transfer draws a structural line between open-weight models and closed proprietary services. The technique requires unrestricted access to tensors in the GPU video memory and integration into the inference engine: the Nvidia paper validates the method in research code, and open-source serving engines are the natural path to implementation, but the approach stays unviable in commercial APIs that expose text interfaces only.

The barrier is physical.

That distinction carries immediate strategic consequences for CTOs and lead engineers. Tensor reuse depends on absolute control over the physical memory buffers on the accelerators. Platforms running open-source serving engines, such as vLLM and SGLang, offer the level of control needed to implement custom modules that extract activation tensors and inject them directly into adjacent processes.

On public cloud APIs with closed models, that level of intervention is impossible. Commercial providers offer prompt caching within their own operational limits, but provide no endpoints for extracting raw attention matrices. An organization cannot pull the KV cache of one provider's model and apply it to a different model.

The economic advantage tilts toward operations that sustain their own clusters or dedicated instances. Companies that keep on-premise infrastructure or reserved cloud instances will be able to orchestrate agent fleets where long triage tasks run on compact models and decision-making happens on heavy models, with no handoff cost.

The role of AI gateways and the positioning of the Nexforce Router

The evolution of agentic workflows requires the infrastructure layer to manage cost and latency together. The Nexforce Router acts as the enterprise gateway for AI models, providing intelligent routing, automatic failover, and per-agent budget control, preparing enterprise architectures for efficient transitions between providers and models.

As techniques like KV cache transfer mature at the inference server level, the need for centralized governance grows more urgent. An intelligent model gateway must decouple the final application from the complexity of multiple providers and open-source variants. The Nexforce Router centralizes that management in a single OpenAI-compatible endpoint.

The platform solves critical pains of AI operations at scale:

• Intent-based intelligent routing: distributes requests based on cost per token, required latency, and context limit, sending simple tasks to low-cost models and complex tasks to frontier models.
• Governance with spend caps: sets strict budget limits per API key or per individual agent, limiting accidental billing spikes in autonomous chains.
• High availability with automatic failover: monitors response health and moves calls within milliseconds when a provider slows down or becomes unstable.

Unified log and metric management in the Nexforce Router gives exact visibility over every token consumed in corporate operations. Traceability is indispensable. It reveals which agent lifecycle bottlenecks make transitions between analysis and production tasks expensive.

Infrastructure cannot be hostage to rigid architectures. Enterprise gateways guarantee the company captures cost reductions in real time, without rewriting product code at every innovation in hardware or tensors.

Practical decisions for software architects and engineering leaders

To capture efficiency gains in agentic systems before cross-model caches are fully standardized, engineering teams should structure their pipelines in clear steps. Separating tasks by complexity, optimizing context retention at the gateway, and standardizing inference interfaces are immediate measures to contain spend and cut operational latency.

The following actions direct the modernization of workflows with a focus on operational sustainability and economic efficiency:

1. Map the handoff tax in production pipelines: measure exactly how many tokens are reprocessed at every agent switch and calculate how much GPU time goes into rebuilding attention in later steps.
2. Split agents between explorers and synthesizers: assign lighter models to sweep databases, reserving calls to the most expensive models strictly for the final analytical synthesis.
3. Centralize quotas and routing policies at the gateway: use the Nexforce Router to set consumption ceilings per agent and make sure no failure in autonomous chains hits the monthly AI budget.
4. Build dedicated inference environments for latency-sensitive workloads: evaluate high-efficiency engines such as vLLM on private clusters where access to memory tensors allows testing linear cache projection pipelines.
5. Isolate application code from proprietary contracts: adopt agnostic connection patterns so model swaps and flow redirection happen through route configuration, with no source code refactoring.

Implementing these measures creates immediate operational discipline. Teams that control the route of their tokens harvest substantial savings today and prepare their systems to absorb physical memory transfer tomorrow.

FAQ on KV cache in AI agents

Questions about how the KV cache works and its impact on multi-model flows involve GPU memory concepts, architecture limits, and practical viability. The answers below clarify the role of prefill, applicability on public APIs, and the difference between conventional caching and direct transfer between models. Four questions cover the recurring ground.

What exactly is the KV cache and why does it consume so much GPU memory? The KV cache stores the key (Key) and value (Value) tensors computed by the attention layers of a Transformer model for every previous token in a conversation. As the context grows, this history requires substantial VRAM allocation to avoid recalculating attention at every new token generated.

Does KV cache transfer between models work on APIs such as OpenAI's or Anthropic's? No. Closed model providers deliver text interfaces over the network only and grant no access to tensor buffers in GPU memory. Physical KV cache transfer requires total control over the inference servers, working exclusively on instances with open-weight models.

What is the difference between traditional prompt caching and cross-model KV cache transfer? Traditional prompt caching stores and reuses the KV cache only within the same model and instance, when text prefixes match. Cross-model transfer mathematically converts the cache generated by a smaller model so a different, larger model continues the reasoning without recomputation.

Does ridge regression tensor projection degrade the final model's answer quality? Nvidia's experiments recorded 97.6% average accuracy retention of the receiving model on the Qwen3 14B to Qwen3 32B transition, and retention ranges from 73% to 98% across the four working pairs. Small losses can occur on tasks of extreme sensitivity, and pairs outside those conditions can degrade heavily, but the speedup of up to 25x pays for the variation in the vast majority of corporate applications.

References and Further Reading

• Heo, Taekyung et al. (August 4, 2026). Cross-Model KV Cache Transfer in LLM Families: A Closed-Form Linear Mapping for Prefill Reuse. NVIDIA. arXiv:2608.03893. Available at: https://arxiv.org/abs/2608.03893
• Fu, Tianyu et al. (2025, revised March 2026). Cache-to-Cache: Direct Semantic Communication Between Large Language Models. arXiv:2510.03215. Available at: https://arxiv.org/abs/2510.03215
• Nexforce Router (2026). Technical Documentation on Intelligent Routing and LLM Cost Management. Available at: https://nexforce.ai/router

What comes next for KV cache transfer

Over the next few months, the work moves from research papers into serving engines. Today's method works within a model family with compatible cache formats: 73% to 98% accuracy retention on the working pairs, heavy degradation on the two Ministral pairs, and 24 to 37 points recovered by the nonlinear MLP mapper. The support lives at the inference-serving layer, and vLLM and SGLang provide the control points, so adoption tracks how that integration matures. The Nexforce Router does not transfer cache. It decides which model runs each task and caps spend per API key and per agent. Once a handoff stops paying prefill twice, the routing layer is what captures that saving at fleet scale.