CUDA and ROCm: The GPU software layer that decides which chips AI can run on

CUDA and ROCm are the software layers that sit between an AI model and a graphics processing unit (GPU). CUDA belongs to Nvidia. ROCm is AMD’s open counterpart. A GPU is useless to AI developers until this layer exists, which is why the software can matter as much as the chip.

The topic surfaced in Anthropic’s IPO coverage. Our report on the reportedly oversubscribed offering notes that executives showed how the company uses its own models, including a claim that Claude was put on AMD chips in three days.

A GPU speaks its own language, and someone has to translate

Think of a GPU as a huge kitchen with thousands of cooks. Each cook is fast at one simple job, and the kitchen does well only if every cook gets precise instructions. A language model needs billions of small multiplications, which suits that kitchen well.

Software has to write those instructions. A CPU (the computer’s general-purpose processor) cannot do that for a GPU on its own. CUDA and ROCm supply the toolkit: compilers, a runtime, and libraries of ready-made math routines.

Kernels: The small programs that run thousands of times at once

The core unit is the kernel, a function that runs on the GPU. Per the CUDA description on Wikipedia, the CPU (the host) sends data to the GPU (the device), launches a kernel, and copies results back. Many copies of the kernel run in parallel.

Each copy is a thread, and a thread usually handles one piece of the data. Threads are grouped into blocks, and blocks form a grid. Groups of 32 threads, called warps, run the same instruction together.

AI frameworks such as PyTorch hide most of this. Underneath, they call tuned kernels from libraries. Fast AI code depends on the quality of those kernels for each specific chip.

Two stacks, one lock-in problem

Nvidia created CUDA in 2004 and released it officially in 2007. Ian Buck, who had built a language called Brook as a Stanford PhD student, and John Nickolls turned Brook into CUDA after Nvidia hired Buck. Nvidia’s page describes it as a platform for accelerated computing with libraries, debugging and optimization tools, a C++ compiler and a runtime library. CUDA-enabled GPUs come only from Nvidia.

AMD’s ROCm, per AMD’s documentation, is an open software stack for GPU-accelerated computing. It includes the compilers and runtimes (HIP and LLVM-based tools), math libraries, communication libraries, profilers and debuggers. AMD lists PyTorch as an example of a supported deep learning framework.

Years of CUDA-only code and expertise are what people mean by a software moat. Moving to another chip means moving the kernels too.

Why porting is slow, and what Anthropic claims to have changed

AMD offers HIP, a C++ API that CUDA developers will find familiar, plus HIPIFY tools that convert CUDA calls into HIP calls. AMD’s own documentation says HIP is not intended to be a drop-in replacement for CUDA, so porting usually needs manual coding and performance tuning. Translation layers such as ZLUDA and SCALE also exist, according to Wikipedia.

Writing and tuning kernels for a new chip is the expensive part. Secondary coverage of Anthropic’s roadshow, which we could not view directly, reports a slide comparing about three days with its Claude Mythos model against six to 12 months with the traditional approach. OfficeChai reports that the earlier process took dozens of engineers months, and that the model wrote or ported almost all of the kernels with little human intervention.

Reports describe a narrow scope. The coverage says the three days covered prompt processing for one Anthropic model on AMD MI355X GPUs at a useful level of performance. One outlet cautions that this does not show any model can move between GPU platforms in three days. Anthropic’s exact wording remains unverified here.

The AMD deal that gives the software work a purpose

The Decoder reports that Anthropic will deploy up to 2 gigawatts of AMD Instinct MI450 GPUs in Helios servers, with the first gigawatt phase starting in the first half of 2027. AMD will invest up to $5 billion in Anthropic. Anthropic already runs MI355X GPUs, and the companies plan a multi-year effort to use Claude to improve ROCm and GPU workloads.

The same report notes that critics question whether such circular deals, where suppliers fund the labs that buy from them, are sustainable. That is a financial debate and separate from the technical one.

Limits, and what to watch

Working is not the same as fast. A port that runs at a useful level can still trail a hand-tuned CUDA version. Performance per dollar decides whether a lab adopts a chip at scale.

Newer hardware also needs its own kernels. The three-day example used MI355X GPUs, while the supply deal covers the newer MI450 series.

Three things are worth watching. Whether independent engineers can reproduce AI-written kernels on other chips. Whether ROCm closes the library and tooling gap with CUDA. And whether cross-vendor efforts, such as the UXL consortium that Wikipedia lists as working on an open alternative to CUDA, gain traction.

If AI can cut the cost of writing kernels, the lock-in from CUDA gets weaker. The claim is attributed to Anthropic’s roadshow, and the evidence so far is a single reported example.