Google brings vLLM and SGLang to TPUs in PyTorch, with Meta already running it

Google has launched native PyTorch versions of the vLLM and SGLang inference engines for its TPUs, built on a new layer called TorchTPU. According to the Google Developers Blog, SGLang-TorchTPU already runs in production at Meta.

PyTorch code now runs on TPUs without JAX

Both frameworks already ran on TPUs through JAX-based backends. Google says developers kept asking for a way to customize and debug their serving stack in PyTorch, so it built TorchTPU, which treats the TPU as an ordinary PyTorch device. Tensors are standard PyTorch tensors with no wrapper types, and code runs eagerly or compiles with torch.compile through XLA. Collective operations go through torch.distributed.

Each framework keeps its own scheduler, KV cache manager, worker model and API server. Google adds TPU implementations underneath, plus custom kernels written in Pallas that PyTorch calls as custom operators. Most models reuse the upstream PyTorch definitions from vLLM or SGLang.

The feature list covers automatic prefix caching, KV cache offloading to host memory and Managed Lustre, speculative decoding with several drafters (including EAGLE-3 and n-gram), disaggregated prefill and decode, and multimodal models. It also lists FP8, FP4 (MXFP4 and NVFP4) and INT4 checkpoints, and parallelism across hosts. For KV cache transfers, both engines use TPU Sync, an open-source C++ library that reads PyTorch device buffers directly and moves data on its own threads.

Google lists Qwen, DeepSeek, GLM, Kimi, GPT-OSS and Gemma4 among the supported model families. The earlier vLLM-Jax and SGLang-JAX runtimes will stay maintained. Code is on GitHub under vllm-project/vllm-torchtpu, sgl-project/sglang-torchtpu and google-pytorch/torch_tpu. The post gives no pricing for the engines, which are open source, and does not say which Google Cloud regions offer the hardware.

Cost and speed claims come from Google and its partners

Google says both engines are running at scale with preview customers. Its public evidence comes from the SemiAnalysis InferenceX benchmark: on Qwen 3.5 with Ironwood TPUs (TPU 7x), it reports the lowest cost per million tokens overall at 2.01 cents in high-concurrency configurations. It also says it published new results on AgentX, the project’s track for long-context and multi-turn agent workloads.

That Kimi K3 figure is Google’s description of work with Inferact, which also led a preliminary AgentX submission. The post gives no hardware configuration or comparison systems for it, and none of these numbers have been independently checked here. Zongwei Zhou, PyTorch-TPU lead at Meta, said in the post that SGLang-TorchTPU “is running in production at Meta today”.

Vendor-chosen benchmark settings matter. SemiAnalysis’s first InferenceX results for Ironwood, dated September 7, 2026, were summarized by secondary outlets as $0.181 per million tokens for Ironwood against $0.222 for Nvidia’s B200 at 100 tokens per second per user on FP8 models. That is a different setup from Google’s 2.01 cents, so the two figures cannot be compared directly. The same coverage said Ironwood did not win every comparison.

A bet on open software to widen the TPU audience

Analysis: Strategy. TPUs have long been tied to JAX, while much of the open model ecosystem is written for PyTorch and Nvidia GPUs. Letting vLLM and SGLang run natively on TPUs lowers the cost of trying the chips for teams that already use those engines. The partnerships point the same way. Inferact, founded by vLLM’s creators, launched in January 2026 with $150 million in seed funding at an $800 million valuation, according to Pulse 2. RadixArk is led by SGLang’s creators. Google also says vLLM is launching a free TPU capacity pool for developers who contribute to both engines.

The timing fits a crowded week for inference software. We recently covered llm-d 1.0 and its self-hosted inference pitch and DeepSeek’s open-sourced DeepInfer stack. Google’s pitch differs by tying the software to its own silicon, and its advantages are cost per token and the hardware it controls.

Several gaps remain. SemiAnalysis said earlier that the TorchTPU stack was expected to leave private beta and be open sourced around October, which matches this launch. Whether outside teams can match Meta’s results, and how the engines compare with mature GPU deployments on other models, is not established. Google says more optimizations and Day 0 support for new open models are under way, and it invites feature requests through GitHub issues.