Deepgram’s Nova-3 now runs on Trainium2 and gets up to 70% more audio per dollar

Amazon and voice AI company Deepgram have moved the Nova-3 speech-to-text model onto AWS Trainium2 chips. According to an AWS blog post, offline batch jobs process 11 to 70 percent more audio per dollar than the compute Deepgram used before.

Eight model copies on one small Trainium2 instance

The work targets offline, throughput-oriented transcription such as contact-center analytics and medical transcription. It does not cover live streaming. Deepgram handed over Nova-3 as standard PyTorch code, and the AWS Forward Deployed Engineering team spent several months porting and tuning it with Deepgram. The final setup runs eight model replicas on a single trn2.3xlarge instance.

The team used TorchNeuron, the PyTorch integration of the Neuron compiler. After a first pass in eager mode to find operations that fell back to the CPU, they compiled the encoder and decoder as separate static-shape graphs with torch.compile. AWS says no CPU fallbacks remained. Accuracy was checked against CPU output layer by layer, and end-to-end word error rate had to match Deepgram’s reference values.

Tuning then moved to kernels. Pre-built kernels from the public NKI Library covered fused attention projections, feed-forward layers and depthwise convolution. Profiling with Neuron Explorer showed where hand-written replacements would do better. AWS reports these gains:

• A rewritten feed-forward kernel is 2.0x faster than the reference at batch size 96, because it loads each weight tile once and reuses it.
• A depthwise convolution kernel moved to the Vector Engine and is 1.5x faster at batch size 96.
• A new two-dimensional convolution kernel removed a compilation barrier above batch size 32, and AWS reports up to 50x on the convolution module at batch sizes of 32 or less.

Deepgram’s serving stack is written in Rust, so the model also had to run without Python in the serving path. The team built a runtime in which Rust threads pull work from a shared queue, one per replica. A C layer loads precompiled Neuron executables through the Neuron Runtime API. To fit eight replicas on one device, they switched each logical core to a single physical NeuronCore. AWS says the runtime reproduced the reference token sequences exactly at every tested batch size, and beat a multiprocessing Python baseline by 7.5 percent at the throughput-maximizing batch size.

What the 11-70 percent range does not tell you

The headline figure is a wide range, and the post does not say which workload or which baseline produced the low end or the high end. It also does not name the alternative hardware, so readers cannot tell whether the comparison is against GPU instances. AWS says it used instance pricing at the time of the evaluation, and reports throughput only as relative gain, not as absolute audio per second. Nothing here is independently verified: the post is written by AWS and describes work done with Deepgram.

For customers, nothing changes immediately. The post says the gains help Deepgram scale and offer lower prices, and that benefits could extend to customers who self-host Deepgram models in their own VPC on Amazon SageMaker AI. It announces no price cut, no new Deepgram tier and no date. Developers who want to repeat the approach have documentation links but no packaged Nova-3 build for Trainium.

Why a hand-written kernel still matters

The post is a useful look at how far a team gets with off-the-shelf parts and where it still has to write low-level code. Pre-built kernels got the team most of the way. Three layers needed custom work because the generic versions either wasted memory bandwidth, used the wrong compute engine or blocked compilation at larger batch sizes. Speech models run many small decoder calls per audio chunk, so the cost of dispatching work from the host becomes the bottleneck at low batch sizes. That explains why the Rust runtime helps most there.

The context is Amazon’s push to make Trainium easier to adopt. A Trn2 instance family has 16 Trainium2 chips with 1.5 TB of HBM3 memory, according to AWS’s instance page. At launch, Amazon said Trn2 offers 30 to 40 percent better price performance than the then-current GPU-based EC2 instances. The Deepgram range is broader than that earlier claim, but it comes from a single workload.

The same day, Amazon also presented TorchNeuron as a way to run existing PyTorch code on Trainium, a launch we covered separately. Amazon said it would open-source the software under Apache 2.0 on October 20, 2026. It named voice AI company Cartesia as an early user. The Deepgram project shows the other side of that pitch: compiling the model was the easy part, and the savings came from months of kernel and runtime work done with AWS engineers.

Whether others see similar gains will depend on their models and batch sizes. The post’s closing advice is to start from the Neuron documentation and the NKI samples repository, and to contact AWS Forward Deployed Engineering for help scoping a similar port. Pricing for the trn2.3xlarge instance is listed on the Amazon EC2 Trn2 page.