A 2.8T MoE Model in Production Requires Coordinated Serving

The 2.8 trillion parameter number seems like a memory problem. In serving, it is a coordination problem. The Kimi K3, presented in July 2026, activates around 104 billion parameters per token, selects 16 out of 896 experts and still needs to maintain two types of attention states. The hard part is not in the headline. It's in the path of every token.

The Kimi K3 technical announcement, published in July 2026, explains the architecture. The technical report on arXiv details the scale. What these documents alone don't solve is the operational question: how do you transform an architecture of this size into a service that preserves throughput, latency, and utilization without treating GPU memory like an infinite closet?

The thesis is simple: large MoE models reach production when the infrastructure transforms full scale into active compute, reusable memory, and distributed traffic. Opening the weights is an availability decision. Making them serviceable is an architectural decision.

What changes when serving a 2.8T MoE model in production?

Serving a 2.8T MoE model changes the bottleneck from an isolated multiplication to a chain of memory, communication, and scheduling. Active experts reduce the calculation per token, while the distribution between accelerators, buffer occupancy and cache consistency decide the real capacity of the service.

In a dense model, each token goes through practically the same set of weights. In a MoE model, the token passes through an expert router, which chooses the experts responsible for the next part of the calculation. In the technical case of the Kimi K3, there are 16 experts routed among 896, in addition to the shared components.

This sparsity explains how such a high total scale can exist without each token paying the full bill. It also creates the first pitfall: a silent GPU next to a congested one is not an implementation detail. It's lost throughput.

The load is not distributed perfectly evenly. A programming prompt, a long context request, and a short question can produce different selection patterns. If many tokens look for the same experts, queues and traffic between ranks arise even when the cluster average seems comfortable. The GPU average is a photograph taken from too far away.

Monitoring needs to observe, at a minimum, usage per expert, dispatch time, traffic between ranks, memory occupancy, tokens processed, time to first token and time between tokens. The architecture is no longer a box that receives text and returns text. It becomes a small stock exchange, with very different assets competing for the same liquidity.

The serving preview published by vLLM summarizes the decision set: 2.8T parameters, 1 million token context, 69 KDA layers, 24 MLA layers and 896 routed experts. Each number breaks a common serving hypothesis. Together, they break several at the same time.

Why does a 2.8T MoE model need two types of memory states?

A 2.8T MoE model may require two types of state because Kimi Delta Attention, or KDA, keeps an updated recurring state in place, while Multi-head Latent Attention, or MLA, keeps a KV cache associated with context positions. A state is mutable. The other grows with the tokens and can be paged.

The difference seems academic until the first shared prefix. In MLA, the serving stores keys and values of blocks already processed. A new request that shares the prefix reuses these blocks without recalculating everything. The KV cache is append-only: the past does not change when the next token arrives.

The KDA state is another creature. Each layer maintains a recurring structure that is overwritten with each token. A shared state cannot be directly handed over to two continuations that will update it. Before moving forward, the server needs to restore the checkpoint to a private slot. Then, you need to capture the new state in a safe point so that another request can reuse it.

The operational problem has a name: copy-on-write. The server shares the checkpoint until the moment a sequence diverges, copies the state to its own area and allows mutation. Next, the server needs to preserve a private copy of the state before allowing new mutations. The technical material from SGLang and Miles describes the combined management of these states, with safe reuse, unified memory and phase parallelism.

The distinction changes memory accounting. Weights, KDA state and MLA KV cache do not have the same life cycle. Reserving a single block called “model memory” is the quickest way to find out, in production, that the model fits in the cluster, but the second request does not fit in the replica.

How do KDA state and KV cache of MLA change serving?

The scheduler needs to align two different physical states on the same logical prefix boundary. KDA state is mutable and requires private copying before continuation. The MLA KV cache grows with the tokens and can be paged. This difference determines when a request reuses work and when it pays for memory and prefills again.

The point is simple. The cache is not a single block.

Traditional prefix caching starts from a simple unit: complete blocks of tokens. For KDA, storing a large state in each small block costs too much memory. Saving only in large blocks saves space, but reduces the points at which a prefix can be reused. Two requests that share almost the entire prompt may diverge before the physical boundary and lose valuable reuse.

The implementation described in preview of support for Kimi K3 in vLLM separates three decisions: the physical size of the state block, the alignment required by the scheduler, and the granularity used to identify the prefix. The state can occupy a larger physical block, while the hash recognizes a finer boundary.

When there is a partial prefix-cache hit, the corresponding KDA state needs to be copied to a private destination before continuation can proceed. The KV cache can continue to be shared until new tokens are written. This copy is not a cache miss. It's the right price to preserve a mutable structure without corrupting the prefix of another request.

Caching also changes the economics of prefill. In a coding load, in which several requests reuse instructions, tools and history, a hit on the prefix avoids recalculating the most expensive part of the prompt. Kimi's announcement reports over 90% cache hits on the official API for coding workloads. That number belongs to that architecture and that workload. It is not a rate an operator can assume for arbitrary traffic.

What does LatentMoE save and what does it make harder?

Kimi K3 uses Stable LatentMoE to route 16 of 896 experts in a 3,584-dimensional latent space, according to the SGLang technical description. The reading that this smaller representation can reduce the cost of moving and processing experts is an architectural inference, not a cost result measured in sources. Sparsity cuts calculation. It does not eliminate the need to put the right expert on the right accelerator at the right time.

Kimi K3 combines 896 experts and 16 active experts per token. The execution needs to calculate scores, select experts, group tokens and dispatch them to the ranks that have the corresponding weights. Then, it needs to gather the outputs and restore the order expected by the rest of the network.

This cycle is sensitive to the batch format. In small batch, launching hundreds of small kernels can cost more than arithmetic. In a large batch, the problem changes: communication and balancing begin to dominate. The job of serving stops being “using the GPU” and becomes not creating a procession of microtasks that arrive late to the next collective.

MXFP4 quantization helps keep weights within a manageable memory footprint. The technical announcement also describes MXFP8 activations and quantization-aware training. The deployment configuration published by AWS shows a reference route with MXFP4 weights, tensor parallelism across eight accelerators, and a compatible MoE backend.

This documents a configuration. It does not transform all infrastructure into a universal formula.

Quantization is not synonymous with free memory. Compressed weights share space with runtime copies, temporary buffers, activations, CUDA graphs, KDA state, MLA KV cache and communication area. A server can load the weights and still fail to admit a new long context string. The first number says the model fits. The second tells how many requests fit into it.

Where do prefill, decode and parallelism diverge?

Prefill and decode press different parts of the serving. Prefill processes many tokens and favors large batches, chunks and communication overlap. Decode repeats small steps, sensitive to latency, kernel launches, recurrent state and cache capacity per request. Parallelism needs to respect this asymmetry.

They are different phases.

The parallelism architecture needs to respect this difference. Using the same configuration for both phases because the deployment file accepts a single value is an administrative decision, not a performance decision.

1. Prefill: divides the prompt into chunks and keeps the steps busy, hiding the transfer between stages behind the calculation of the next chunk.
2. Decode: preserves the KDA state, accesses the MLA KV cache and reduces the time of each step. Aggregated throughput is not enough.
3. Expert parallelism: distributes experts by ranks and pays for communication to send tokens to the responsible expert, with gains when the balance exceeds this cost.
4. Tensor parallelism: on Kimi K3, it does not fragment the MLA KV cache because there is a single KV head; each rank maintains a complete copy, while GEMMs are divided into eight and pay a collective per layer, according to the SGLang study.
5. Serving disaggregated: separates prefill and decode workers, allowing each group to scale to the traffic profile it serves.

The SGLang study on Kimi K3 support describes prefill with parallel pipeline in chunks and decode with context parallelism. It also records a disaggregated configuration that achieved 2,808 tokens per second per GPU with chunk-parallel pipeline, context parallelism and the topology described in the study.

The number is a measurement of that topology, hardware and protocol. It's not a promise for any cluster.

Aggregated throughput can hide a bad user experience. A cluster can produce many tokens per second and still deliver the first token slowly if the prefill is congested. It can also have good TTFT and slow decode, turning a long response into a successful-looking queue.

How much infrastructure is needed to host the weights?

The infrastructure depends on the format of the weights, the number of replicas, the supported context, the cache and the communication topology. Sources confirm deployment scale with MXFP4 and eight accelerators on a reference instance. This data delimits a documented configuration, not a fixed figure for every workload.

The calculation starts with the weights.

Capacity needs to be read on six chained envelopes. Each limits the next, and the slack disappears when traffic combines long context, concurrency, and inter-rank communication.

1. Weights: compressed copy of the total model, distributed according to the parallelism strategy.
2. Runtime: kernel buffers, workspace, graphics and communication areas.
3. State per request: KDA state, which grows with the number of active sequences.

The first three tell you what it costs to deploy the model. The next three determine its production serving capacity.

1. Context: MLA KV cache, which grows with stored tokens and batch size.
2. Replication: additional copies for availability, regions, peaks or workload isolation.
3. Transfer: internal network used in tensor parallelism, expert parallelism, cache transfer and serving disaggregated.

The sum defines the limit for admitting context and concurrency. If the operation reserves almost all memory for the long cache, it lacks space for the state of new sequences. If you reserve almost everything for KDA states, the MLA KV cache becomes the ceiling.

The unified memory proposal presented in the SGLang material attempts to reduce this bet: a single pool allows KDA states and MLA blocks to occupy capacity as the workload changes. This is capacity management, not hardware magic. Unified memory doesn't reduce weights and doesn't make a GPU fit where it doesn't fit. It just prevents one pool from becoming full while unused bytes remain trapped in the other.

The figure of approximately 5 TB is not included in this analysis as a fact. The briefing's list of sources does not support a verifiable decomposition of this total between weights, runtime, states, replicas and communication. Without this decomposition, the number impresses more than it informs.

What does this architecture change for model routing?

For a routing layer, the destination capacity becomes an operational variable. The router does not host the Kimi K3, does not control its kernels and does not manage the physical memory of the GPUs. It can route each request according to cost, latency, context, performance, availability and observed load. The endpoint architecture now informs the traffic decision.

The model name is not enough.

A request with a long prefix and high probability of reuse does not have the same operational cost as a short question without cache. A request that requires the context of 1 million tokens should not compete for the same path as a simple request just because they both use the same model name. The model name is insufficient information for the decision.

The Nexforce Router acts as a gateway and routing layer between applications and models. Its rules can consider cost, performance, latency and context, distribute load, apply limits per key and failover. The boundary is important: the Router governs the traffic arriving at an endpoint. The endpoint vendor or operator remains responsible for the physical serving architecture.

A routing policy for large models needs to observe four signals before forwarding:

1. Eligibility: Does the target support the required context window, modality, and output format?
2. Probable state: does the request have a reusable prefix or does it arrive as a cache miss that will require a complete prefill?
3. Load: does the destination have decode capacity, memory for new states and margin for the current batch?
4. Economics: Does the quality gain justify the cost of activating a large footprint route when latency increases?

Lowest price routing alone is a trap. The cheapest model per token may produce more tokens, lose the prefix, suffer from an expert queue, or return a response that is too slow for the SLA. The variable that matters is the cost per result within the latency contract, not the isolated price in the table.

The comparison of LLM costs in 2026 becomes more accurate when the serving cost is included in the account. The decision does not choose the model for the lowest price per token. It asks which destination delivers the required result with the acceptable combination of cache, latency, load, and quality.

What are the limitations of the analysis?

The analysis separates three layers: confirmed architectural facts, documented infrastructure configurations, and routing implications. The sources support the KDA and MLA architecture, the Kimi K3 numbers and the serving paths described. They do not support a universal cost, a fixed capacity per replica, or an SLA for any cluster.

The border matters.

The following are confirmed: 2.8T of parameters, around 104B active, 896 experts, 16 active experts per token, context of 1 million tokens, combination of KDA and MLA, MXFP4 weights and specific changes to the cache and kernels. Also documented are the challenges of mutable state, unified memory, phase parallelism and serving disaggregated.

It is not confirmed by a single universal source: the exact cost for each company, the final number of GPUs for each SLA, the cache rate in a non-coding workload, the average use of experts in their own traffic or the capacity per replica in other hardware. These numbers require benchmarking with actual prompts, lengths, concurrency, and SLOs.

The editorial conclusion is narrower and more useful: the larger the sparse model, the less sense it makes to measure serving just by the number of parameters or the price per token. The system needs to measure state, cache, communication, phase and destination.

The best argument in favor of open MoE is also conditional

The strongest argument in favor of serving an open MoE model is straightforward: the company gains control over the placement of the model, can adapt the serving stack to its own workload, and stops depending on an external capacity whose price, availability and policy can change. Sparse activation makes full scale less daunting per token. Quantization reduces the pressure of the weights. The cache turns repeated prefixes into already paid work.

This argument is solid. It would be a mistake to treat it as a fantasy.

The problem is that control does not eliminate costs. It redistributes those costs. The company starts paying for expert distribution, interconnection, kernel engineering, memory-pool design, observability and the idle capacity necessary to survive spikes. A model that looks cheap on the weights sheet can become expensive when the replica needs to maintain long context, failover reserves, and margin for congested prefill.

The conditional conclusion is this: open weights are an infrastructure advantage when the workload has enough volume, prefix repetition, and control requirements to pay for the complexity. For irregular or low-use traffic, the same freedom can turn into idle capacity.

The mature decision does not ask whether MoE is cheap. It asks what part of the account the operation is willing to own.

FAQ about 2.8T MoE model in production

A 2.8T MoE model does not apply all parameters to each token, but the operation still needs to distribute the total weights and coordinate experts, KDA state, MLA KV cache, runtime memory and communication. The throughput decision depends on the load, topology, and latency contract, not the headline number alone.

Does a 2.8T MoE model use 2.8T parameters in each token?

No. In the technical case of Kimi K3, 16 out of 896 routed experts are token activated, with around 104 billion active parameters. The total weights still need to be distributed and kept available so that the router can choose the correct experts.

Does KDA completely replace the MLA KV cache?

No. KDA and MLA have different roles and states in the hybrid architecture. The serving needs to keep the KDA recurrent state and the MLA KV cache aligned on the same logical prefix boundary.

Does Prefix caching work the same way in KDA and MLA?

No. The MLA KV cache is append-only and can be paged by blocks of tokens. The KDA state is mutable and requires checkpoints, copy-on-write, and ordered copies before continuation changes the shared state.

Does MXFP4 solve the memory problem?

MXFP4 solves part of the weights problem, but does not close the memory count. The serving also reserves runtime, KDA state, MLA KV cache, buffers, replicas and communication. Therefore, successful model loading does not prove that the replica sustains the expected concurrency.

Weight is just the input.

Does the Nexforce Router host the Kimi K3?

No. Nexforce Router is a gateway and routing layer. It distributes requests between destinations and can apply cost, latency, context, load and failover rules, without controlling kernels or physical memory of the serving.

References and Further Reading

• Kimi K3: Open Frontier Intelligence, Moonshot AI technical announcement, July 2026.
• Kimi K3: Open Frontier Intelligence, technical report, submitted July 27, 2026.
• A Preview of Production-Scale Kimi K3 Support on vLLM, July 22, 2026.
• SGLang and Miles Add Day-0 Support for Kimi K3, July 27, 2026.
• Deploying Kimi K3 on Amazon SageMaker HyperPod and Amazon EKS, July 30, 2026.
• Kimi K3 repository, implementation and deployment files.

The next serving evaluation

The next evaluation of a model of this size should not start by asking how many parameters it has. It should ask how many states fit, how many prefixes are reused and how much of the communication is on the critical path. The scale is impressive in presentation. The serving decides whether it becomes a product or just a very expensive photograph of a busy GPU.