DeepInfer: DeepSeek's open-source inference engine

DeepInfer is the inference engine DeepSeek announced on 20 October 2026 under Apache 2.0, and it runs a single model, DeepSeek-V4.1-Flash. The package ships kernels and parallelism. It does not ship the layer that turns an engine into a service.

The official repository, read on 22/10/2026 and created on 16 October, ships the engine under Apache 2.0. A separate repository, deepseek-recipe, carries MIT. Under the heading "Required for a complete inference service (not included in this release)" three items: KV cache, the disaggregation between vision encoder, prefill and decode, and balancing across data-parallel replicas.

What came is the engine. What stayed out is the specification of a serving layer.

What DeepInfer delivers

DeepInfer delivers the engine down to kernels and parallelism: FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect, attention in DP with balanced CP, MoE in EP with EPLB, DSpark and structured outputs in regex, JSON schema and EBNF. It is the inference framework, not the service.

One clause prevents confusion: attention in DP with balanced CP is an engine feature, but the Q&A says that, in the service DeepSeek operates with V4.1-Flash, "TP and CP are not used for now". The release enables what its own production does not use.

The scope is narrow. The README lists DeepSeek-V4.1-Flash as the only model, with the DeepSeek-V4.1-Flash-1017 checkpoint. The 1M window and the max_tokens of at least 256K come from the model card.

deepseek-recipe completes the protocol, in Rust with Python bindings and an MIT license, created on 10/09/2026: it translates Messages, Chat Completions and Responses into DeepSeek's format and returns either a full response or a stream.

The published commitment is short: inference code for subsequent models stays open in DeepInfer. The engine that shipped is close to the internal one, not identical: the internal FlashMLA decode kernel is 5% to 10% faster, with at most 5% end to end.

The README declares that DeepInfer was built primarily on vLLM, which changes the fork math.

What the package does not include

The DeepInfer README devotes three items to what is missing, and the deployment Q&A repeats the list: KV cache, the disaggregation between vision encoder, prefill and decode and balancing across replicas. None of it is in the repository.

The cache subsystem occupies about 100 PiB of SSD, with a TTL of 3 days for agent traffic and 7 for chatbot traffic.

The caveat is in the README: the disaggregation between vision encoder, prefill and decode does not come, and the README adds that "most required engine functionality is already implemented". What is missing above all is the deployment layer.

EPLB ships in the package and does expert placement, not request admission across replicas. The Expert-Parallel Load Balancer that the Q&A lists among the four internal balancers is one of the replica balancers that are missing: with #DP = #EP, the EP replica is also the replica that receives requests. Treating why failover is not load balancing and admission across replicas as the same function is the same category error.

The Q&A names five internal components: Prefill Load Balancer, Decode Load Balancer, Vision Encoder Load Balancer, Expert-Parallel Load Balancer and KVCache Storage. And it names a sixth, the Entrypoint, which matches prefixes by hash-chain over metadata against the local kvcache-store before routing.

KV cache transfer between models and agents is the problem this cluster solves. It is data plane.

The number that measures the gap is one sentence from the Q&A: "we reach roughly 80% of a perfectly balanced offline benchmark". It is DeepSeek measuring its own service against the offline chart it publishes, with its own balancers on its own hardware. It is not a third-party floor and not something a gateway recovers.

Image: DeepSeek, DeepInfer documentation (github.com/deepseek-ai/DeepInfer)

Offline is not serving

The DeepInfer README describes the release as "offline inference with the core features needed for serving at scale", and the code confirms it: the examples are inference, benchmark and concurrency sweep scripts. The only module with "server" in the name exposes /profile. It does not serve requests.

The Q&A removes the ambiguity: prefill is sized by non-cached tokens, decode by in-flight requests. A decode engine receives a shutdown notice and has 1 minute to hand off requests and KV cache, and the deployment keeps a redundant quota and forbids scaling down when utilization passes 0.8.

"1P + 1D is deployable", with a ratio of roughly 1:1 and decode recommended at EP64 or above; the smallest efficient unit is 64 prefill GPUs plus 64 decode GPUs. The benchmark uses 4 SM100 GPUs for prefill and 64 for decode: the distance between the two is the distance between a chart and an invoice. One failed GPU makes the whole instance unusable, and the choice of open weights against hosted models changes with it.

The numbers DeepSeek published, and what they measure

No third party measured the throughput of this engine. The numbers are self-reported by DeepSeek and live in the repository's Pareto charts, transcribed by MarkTechPost on 20/10/2026: above 25,000 output tokens per second per GPU, at EP64DP64, nextn=3, 75% acceptance and 256 requests per rank. The 80% and the 155 trillion come from the Q&A, not from the charts, and so does the DSpark table.

Without those conditions, the number does not exist: it holds for 64 decode GPUs and a 131,072-token context, with close to 100 tokens per second per user and above 300 with 8 requests. On Ascend there is no measurement: the benchmarks are on SM100.

The only dollar figure is of another nature: one cluster serves 155 trillion tokens per day, with revenue of US$38 per GPU-hour at off-peak pricing, the "valley pricing" of the Q&A read on 22/10/2026. Revenue is not cost. Off-peak pricing is not list price, and an open framework has no price per token. Anyone who wants to cut inference cost needs their own measured load.

Two machines, two feature sets

DeepInfer covers two platforms and no third one: NVIDIA SM100 and Huawei Ascend 950 NPU, with a build on Linux, GCC 13 or higher, Python 3.12, CUDA 13 on one path and CANN 9.2 on the other. Earlier NVIDIA generations, Hopper included, stay out.

Parity stops at the build. Batch-invariant inference and deterministic sampling ship marked "(NVIDIA only)", with their own environment variables and a fixed seed. On the Ascend path neither exists, and production parity is a claim the source does not support.

The number also exists on one side only: 25,000 tokens per GPU and 80% of the benchmark come from SM100, and the Ascend path delivers a build, not a curve. Day-one parity there means running V4.1-Flash.

What the operator still has to build or buy

DeepInfer needs three layers, not two: model, cluster and gateway. An LLM gateway covers model, provider and endpoint choice, failover, spend cap, trace and the choice of the replica with the least work in flight. It does not cover 100 PiB of storage, the P/D fabric, admission inside the engine or a drain that hands off cache.

The trade-off settles the question. The Q&A writes that "since any engine can hit any prefix, there is no prefix-affinity routing", because the 3FS tier makes any prefix reachable. Affinity without a shared tier is a worse deal than pulling by quota.

Five pieces, each with an owner and a cost:

1. KV cache tier: 3FS cluster, TTL of 3 and 7 days. Owner: platform and storage. Getting it wrong costs repeated prefill.
2. P/D fabric: moves KV between stages at 400 Gbps per 8 GPUs. Owner: platform and network. Getting it wrong merges the stages onto the same machine.
3. Admission across replicas: reads the engine's real queue. Owner: serving. Getting it wrong costs idle GPU.
4. Drain: 1-minute notice with KV handoff. Owner: platform. Getting it wrong costs a lost request.
5. Quota and EP size: redundant capacity and EP redefined after a failure. Owner: SRE. Getting it wrong costs the whole instance.

Nexforce Router belongs here as a category: a control plane, it routes across models and providers, and it does not hold the engine's cache, does not separate prefill from decode and does not run DeepInfer. The operational cost of an AI gateway in production is the bill for what it covers.

The decision test before the first clone

Four answers decide whether downloading the DeepInfer repository saves work: the hardware floor on your load, who holds the KV cache, who admits requests across replicas and the real cost of keeping a fork. None of the four is in the README.

1. Hardware floor: it is not 4 GPUs, it is the 64 plus 64 unit, or a fraction in 1P+1D.
2. KV cache owner: if it is your cluster, the work is the 100 PiB tier; if it is the provider, you consume an endpoint.
3. Admission across replicas: the release does not bring it; if nobody on the team has built one, it sets the deadline.
4. Fork cost: Apache 2.0 allows closing it, and the bill arrives at the first update to port.

Frequently asked questions

What is DeepInfer?

DeepInfer is the inference engine DeepSeek announced on 20 October 2026 under Apache 2.0, with kernels, parallelism, speculative decoding and structured outputs for DeepSeek-V4.1-Flash. It is the inference framework, not the service: the repository examples are offline scripts, and the serving layer, with KV cache, the disaggregation between vision encoder, prefill and decode and the balancing, stayed out.

Does DeepInfer replace the DeepSeek API?

No. DeepInfer ships the code that runs the model on the operator's hardware; the DeepSeek API delivers inference as a service, with the cluster, the KV cache and the balancers the release does not include. Running the engine means taking on storage, network and request admission, plus the drain the Q&A itself describes. The US$38 per GPU-hour is revenue, not cost.

What is missing to run it in production?

DeepInfer arrives without three items the README lists as necessary for a complete service: KV cache, the disaggregation between vision encoder, prefill and decode and balancing across data-parallel replicas. The deployment Q&A adds autoscaling, redundant quota, drain with KV handoff and the Entrypoint that matches prefixes by hash-chain. Engine, parallelism and EPLB are already in the package; what is missing is the operations layer.

What is the license, and can it be used commercially?

DeepInfer uses Apache 2.0, confirmed in the README and in the GitHub license field, and deepseek-recipe, a separate repository, uses MIT, confirmed in the LICENSE file. Both allow commercial use and modification, with no obligation to publish what is built on top. What the license does not deliver is the missing code: the balancers and the KVCache Storage stayed in DeepSeek's infrastructure.

References

Sources read on 22/10/2026.

• DeepInfer official repository, README and doc/inference_deployment_qa.md, https://github.com/deepseek-ai/DeepInfer, 20/10/2026.
• deepseek-ai/deepseek-recipe, LICENSE (MIT), created on 10/09/2026.
• Model card for DeepSeek-V4.1-Flash-1017, huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017.
• DeepSeek, document on open sourcing the inference engine, April 2025.
• DeepSeek post on X, 20/10/2026, via The New Stack.
• MarkTechPost, DeepSeek Open-Sources DeepInfer, 20/10/2026.
• The New Stack, DeepSeek open-sources the inference engine behind its API, 20/10/2026.

Next step

The test costs less than the fork. Running the same prompt on more than one model, measuring what changes in quality, latency and cost per task, and choosing where each request lands answers in one afternoon what the repository does not answer. Nexforce Router exists for that step, with one API, one key and a spend cap per key.