DeepSeek open-sources DeepInfer, the stack behind its own services, minus key parts

DeepSeek has open-sourced DeepInfer, an inference framework that its README says powers the company’s large-scale production services. Bloomberg reported that the release could make DeepSeek’s technology cheaper and easier for businesses to run. The code is licensed under Apache 2.0.

What the first public release contains

According to the project’s GitHub page, this release covers offline inference and leaves out some serving components. It supports one model, DeepSeek-V4.1-Flash, and targets two kinds of hardware: NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. It requires Linux, GCC 13 or later and Python 3.12, plus CUDA 13 on NVIDIA or CANN 9.2 on Ascend.

The speed features, as the README describes them:

• Optimized kernels built on FlashMLA, DeepGEMM, DeepEP and custom TileLang kernels.
• Attention data parallelism, balanced context parallelism, and mixture-of-experts parallelism with expert load balancing (EPLB).
• Draft-token decoding (called DSpark) with confidence-based verification scheduling, and bounded sliding-window-attention replay to cut prefill work.
• Asynchronous batch scheduling that overlaps CPU work with device execution. It is on by default.

Two further features are aimed at predictability. Batch-invariant inference and deterministic sampling are meant to give identical outputs regardless of batch composition, or for identical inputs and sampling settings. Both work on NVIDIA hardware only. The framework can also constrain generation with regex, JSON schema or EBNF grammars.

DeepSeek lists what is missing: a KV cache system, full prefill/decode disaggregation, and load balancing across data-parallel replicas. The README says the project was built primarily on vLLM, with inspiration from SGLang and TensorRT-LLM. Its published benchmarks used 4 SM100 GPUs for prefill and 64 for decode, but they appear as charts, and DeepSeek’s cost-saving claims have not been independently checked.

A partial release, in line with earlier signals

The omissions fit what DeepSeek has said before. A community index of its open-infra-index repository states that the company would not publish its full internal inference engine as one repository. It gives three reasons: the codebase has diverged from vLLM, it is tightly coupled to internal infrastructure, and maintenance is a burden. DeepSeek’s stated plan was to share optimizations and extract components as standalone libraries. That index is a secondary source, and I could not confirm the wording on DeepSeek’s own pages.

The pattern is consistent with DeepSeek’s Open Source Week in February 2025. It published standalone repositories such as DeepEP, an expert-parallel communication library, and DeepGEMM, an FP8 matrix-multiplication library. DeepInfer builds directly on both, and on FlashMLA. It is the first time those parts appear in a single runnable stack.

Who can use it, and who may hesitate

For developers who already run DeepSeek models on NVIDIA’s newest hardware or on Ascend chips, the main change is access to the scheduling and decoding techniques DeepSeek says it uses itself. Before, they had to assemble them from separate libraries. The Apache 2.0 license allows commercial use, so cloud providers and enterprises can adopt it without negotiating terms.

The limits are real. A single supported model and two hardware families rule out most existing clusters. The missing KV cache system and replica load balancing are parts that large multi-user deployments typically need. Teams would have to supply these themselves or fall back on vLLM, which DeepInfer is largely derived from. Bloomberg’s framing of lower costs is a plausible outcome, but neither the README nor the headline puts a dollar figure on it.

The timing also matters for buyers. Our earlier report on the US order for civilian agencies to drop DeepSeek, and the vendors built on it showed that the company’s software faces scrutiny in some markets. Open code can be audited and self-hosted, which is different from using DeepSeek’s hosted services. The source does not say whether such restrictions would apply to it.

Chip choice is the other thread. Support for Huawei’s Ascend 950 alongside NVIDIA hardware signals that DeepSeek wants its serving stack to run on more than one vendor’s silicon. Competition for inference silicon is heating up elsewhere, as Tenstorrent’s $1.5 billion round at a $6.2 billion valuation shows.

What remains unknown is how much of the production system the released code represents, how the framework performs on workloads other than the benchmark setup DeepSeek chose, and whether the missing serving components will follow later. The repository’s early state, with only a handful of commits, suggests ongoing changes.