The CPU bottleneck in agentic AI: why AI agents keep processors busy while GPUs wait

The CPU bottleneck in agentic AI is the finding that AI agents often wait on ordinary processors (CPUs), not on the graphics chips (GPUs) that run the language model. An agent does more than write text: it searches, runs code and calls software tools. That work runs mostly on CPUs, so it can set the pace of the whole task.

The idea is behind AMD’s new rack design, covered in our report on AMD Helion.

A research assistant who spends most of the day at the library

Picture a research assistant with a very fast brain. You ask a question. The assistant thinks for a few seconds, then walks to the library, searches the catalog, reads three papers, and returns to think again. The thinking is quick. The walking and searching are slow.

An AI agent works the same way. The model is the fast brain, and it runs on a GPU (graphics processing unit). The library trips are tool calls: a web search, a database lookup, a Python script. Those run on the CPU (central processing unit), the general-purpose chip in every server.

Why a chatbot answer and an agent task load the hardware differently

A plain chatbot reply is mostly one job: the model reads a prompt and produces text. That is GPU work almost from start to finish.

An agent runs a loop. It plans a step, calls a tool, reads the result, and decides what to do next. A single request can repeat that loop many times. Each pass adds CPU work for the tool, plus CPU work to coordinate the steps, check results and route requests.

Dan McNamara, AMD’s senior vice president and general manager of Compute and Enterprise AI, put it this way when AMD announced Helion in October 2026: “Agentic AI is not a single workload.” He listed personal agents, enterprise applications, code execution and complex workflows as cases with different compute needs.

What the 2025 measurements found

The main academic reference is a paper first posted to arXiv on November 1, 2025, by Ritik Raj, Souvik Kundu, Ishita Vohra, Hong Wang and Tushar Krishna. It is titled Towards Understanding, Analyzing, and Optimizing Agentic AI Execution: A CPU-Centric Perspective. An earlier version used the shorter title that TrendForce and our news story cite.

The authors profiled five agent setups:

• Haystack, a question-answering system that retrieves documents from a large text collection.
• Toolformer, which solves math problems by calling a calculator service.
• ChemCrow, a chemistry agent that searches scientific literature.
• LangChain, which answers questions using web search and summarization.
• SWE-Agent, a coding agent that runs shell commands and Python.

Their headline result: CPU-based tool processing can take up to 90.6% of total latency, meaning the time from request to final answer. For Haystack, document retrieval alone accounted for 84.5% to 90.6%. At large batch sizes, CPU dynamic energy reached up to 44% of the total, up from about 20% at small batches.

These figures come from specific test systems. Latency and throughput were measured on a 48-core Intel processor with an NVIDIA B200 GPU. Energy was measured on a different machine with an AMD Threadripper PRO and an NVIDIA H200. A different agent or a faster tool would give different shares.

Software fixes versus building more CPU capacity

The paper also looked at what limits throughput, the number of tasks a system finishes per unit of time. Sometimes the GPU is the limit, through memory capacity and bandwidth. In other cases the CPU is, through too many processes competing for cores, cache coherence (keeping cores’ cached copies of data consistent) and synchronization. The authors report that LangChain and SWE-Agent hit a CPU limit at a batch size of 128.

They proposed scheduling changes in response. One splits large batches into smaller ones so CPU and GPU work overlap instead of queuing. Another picks different parallelism methods for CPU-heavy and model-heavy tasks. Names and figures changed between versions, so check the latest version (v3, April 2026) before quoting numbers. The abstract reports up to 1.7x lower median latency for the batching approach in standalone runs.

That is the software route. The hardware route is to supply more CPUs per GPU, which is where industry plans come in.

Who is betting on it, and where the evidence is thin

TrendForce, a market research firm, says AI data centers today run at roughly one CPU for every four to eight GPUs. It expects that to move toward 1:1 to 1:2 as agents spread. TrendForce also relays an estimate from Arm, which sells CPU designs, that data centers need about 30 million CPU cores per gigawatt today and could need about 120 million in the agent era. That is a projection from a vendor with an interest in CPU demand, not a measurement.

AMD’s Helion is the first major rack design we have covered that is built on this premise. Its CPU-only version scales to 32,768 cores of 6th Gen EPYC processors and is meant as dedicated capacity for running agents, linked to separate model inference infrastructure. Partner systems are planned for November 2026, and AMD says it will contribute the specification to the Open Compute Project.

Several caveats apply:

• The 90.6% figure is a maximum across five test setups, not a typical share for every agent.
• Many agent tools wait on networks or outside services. More CPU cores do not speed up a web page that loads slowly.
• Fan-out agents that run many small tasks in parallel benefit from high core counts. Single steps that need fast individual cores benefit less.
• No independent benchmarks, power figures or pricing for Helion have been published.

What to watch as agent workloads grow

The first test is whether real deployments move toward the CPU-to-GPU ratios TrendForce expects, or whether software scheduling closes much of the gap on existing hardware. The second is independent testing of Helion-class racks once partner systems ship, with latency and cost per completed task measured against GPU-heavy setups.

It is also worth watching which tools agents use. If more work moves to services outside the data center, such as external search or APIs, the CPU load shifts with it. The research so far shows that for the agents tested, the processor was a larger part of the wait than most people assumed.