SGLang now runs on AWS Trainium through its usual server interface

Amazon has made SGLang-Neuron publicly available, an out-of-tree plugin that lets the open-source SGLang engine serve models on AWS Trainium chips. According to the AWS announcement, it ships as a Deep Learning Container from RadixArk, built with Annapurna Labs.

A plugin that keeps the SGLang workflow intact

Amazon says teams can serve models and generate reinforcement learning (RL) rollouts on Trainium through the same SGLang server interface and generation APIs they use elsewhere. Setting SGLANG_PLATFORM=neuron when the server starts selects the Trainium backend, which the plugin registers through SGLang’s plugin entry points.

The division of labor is specific. SGLang core still handles the API server, the request lifecycle, RadixAttention and prefix-cache bookkeeping. The plugin supplies the Trainium-specific models, attention, mixture of experts (MoE) and the graph runner. Underneath, TorchNeuron registers Trainium as a native PyTorch device, standard PyTorch operations drive the model path, and kernels written in the Neuron Kernel Interface (NKI) cover performance-critical parts. Everything is compiled with torch.compile. Amazon argues that the out-of-tree design lets Trainium support and SGLang core evolve separately and faster.

The plugin also supports prefill/decode (PD) disaggregation. One server computes the prompt KV cache and passes it to a decode server using NIXL over Elastic Fabric Adapter (EFA), while each side keeps SGLang’s scheduler.

The walkthrough targets Trainium3 and a 120B model

AWS’s example runs on a Trainium3 Amazon EC2 instance, using a container image that AWS publishes in its public Neuron registry. The image is version 0.1.0 and bundles Python 3.14, Neuron SDK 2.33.0 and Ubuntu 26.04. Inside it, the server is launched with python -m sglang.launch_server for openai/gpt-oss-120b, using --tp-size 8, --ep-size 8, an FP8 (fp8_e4m3) KV cache and a 10,240-token context. The post then queries it through the standard /v1/chat/completions endpoint with curl.

For supported models, features and validated configurations, Amazon points readers to the repository documentation and a getting started guide. The version number shows the release is early.

What the announcement leaves unanswered

The post contains no benchmarks. It does not say how throughput, latency or cost per token on Trainium compare with GPUs or with other Neuron serving options, so the economic case rests on Amazon’s general description of Trainium as offering strong economics. It also does not list which models beyond gpt-oss-120b are supported, nor which Trainium generations and instance sizes are validated, nor any regional availability or pricing. Those details sit in the linked documentation, not in the announcement.

Some context helps size the hardware. According to AWS’s Trn3 documentation, each Trainium3 chip carries 144 GB of HBM3e with 4.9 TB/s of memory bandwidth. Those memory figures matter for a 120-billion-parameter MoE model with a large KV cache, though the announcement does not say how many chips the eight-way setup uses.

This is also not the first public sign of the work. The LMSYS team, which stewards SGLang, described an SGLang Trainium plugin beta that was built on TorchNeuron and NKI. It said RadixArk’s Miles training framework used SGLang on Trainium as an external rollout engine and completed a stable training run. The new release moves that path into a packaged container, though the AWS post does not say what changed since the beta.

For teams already serving with SGLang, the practical appeal is a smaller migration: same server flags and API, different chip. Whether that translates into lower bills is not established by the source and would need workload-level testing. Amazon invites users to try their own workloads, contribute through the project repository and send feedback to shape model and feature support.