Llm-d 1.0 pitches self-hosted AI inference at 5 to 10 times lower cost

IBM Research says llm-d 1.0, the open-source, Kubernetes-native inference project it leads with Red Hat and Google, is ready for production. In its announcement, made at the PyTorch Conference North America, IBM says a self-hosted setup ran 3,000 coding agents for 5 to 10 times less than a hosted service.

Flow control and zero-downtime upgrades arrive

llm-d is described as a distributed inference serving network that deploys inference stacks for any model, on any accelerator, in any cloud. It works with inference engines such as vLLM and SGLang. According to IBM, the 1.0 release is about day 2 operations: keeping a system efficient and dependable once real traffic arrives, rather than adding new headline techniques.

The project already offered several serving methods: routing that is aware of prefix caches, offloading of KV cache across GPU, CPU and storage, batch inference optimization, prefill/decode disaggregation, and wide expert parallelism for large mixture-of-experts models. The new release adds the following, as IBM lists them:

• flow control, fairness and prioritization, so bursty workloads do not swamp latency-sensitive services;
• multi-tenant quality of service for workloads that share hardware;
• zero-downtime operations, including upgrades that do not interrupt service;
• agent-aware serving, which keeps multi-turn requests on the same infrastructure so prefix caches stay warm;
• a stable API.

Carlos Costa, a distinguished engineer at IBM Research and a co-founder and lead maintainer, put it this way: “llm-d is ready for day 2 operations.” Stephen Watt of Red Hat’s Office of the CTO said the project “unifies the PyTorch and CNCF ecosystems” by bringing Kubernetes-native orchestration to vLLM and other modern engines.

The lead maintainers are Red Hat, IBM and Google. The project was co-founded last year with CoreWeave and Nvidia, and AMD, Cisco, Hugging Face, Intel, Lambda and Mistral AI contribute. It is hosted by the Cloud Native Computing Foundation (CNCF) as a sandbox project.

A claim of 5 to 10 times lower serving cost

The figure most likely to draw attention is the cost one. IBM and Red Hat say they ran 3,000 software-engineering agents at the same time on a self-hosted deployment, at 5 to 10 times less than the price of serving the same open model as a service. A separate test of an agentic coding workload found that the prefix-aware router captured 97% of the cache reuse that was available.

The economics follow from how agents behave. Coding agents send long, multi-turn requests that repeat large amounts of context, so sending each follow-up to the machine that already holds that context avoids recomputing it. That is the logic behind the agent-aware serving feature, and it is why cache reuse is the metric IBM chose to report.

Several things are not established. The announcement does not name the model, the hardware, the hosted service used as the comparison, or whether staffing and operations costs are counted. The comparison is between self-hosting an open model and paying a provider for that same model, so it says nothing about closed frontier models. Both results come from the project’s own backers and were not independently verified.

Context from outside the announcement helps. According to a CNCF blog post dated March 24, 2026, llm-d joined the foundation as a sandbox project earlier this year, and SDxCentral reports that the project launched in May 2025. A November 2025 community update, published on Substack, still had it at version 0.4. So the project went from 0.4 to a stable 1.0 API in roughly a year.

Who can use it, and what is still unclear

llm-d is open source and aimed at teams that already run Kubernetes and want to serve models on their own infrastructure or in a cloud of their choice. IBM presents it as a vendor-neutral option for organizations that want frontier-class inference without depending on a single model provider, cloud or hardware vendor. The announcement links to documentation, including a page on flow control, and to posts on the agentic test and the IBM and Red Hat demonstration.

The article gives no pricing, licensing terms or installation steps. It also says nothing about commercial support, though Red Hat and IBM are the lead maintainers. Sandbox status is the entry level of the CNCF’s project maturity ladder, which is worth remembering when reading the word production: the 1.0 label comes from the maintainers, not from a foundation graduation.

For readers choosing between a hosted API and their own cluster, the practical change is that the scheduling controls long associated with shared infrastructure, namely priorities, fairness between tenants and upgrades without downtime, are now part of a stable, vendor-neutral stack. Whether the cost gap holds outside IBM and Red Hat’s own test will depend on workloads that resemble theirs: long, repetitive, agent-style sessions.