Prompt caching: How AI APIs reuse repeated text to cut cost and delay

Prompt caching is a way for an AI service to remember the beginning of a request it has already processed. When the next request starts with the same text, the service skips the repeated work. Providers usually charge much less for that reused part, and the answer starts sooner.

Moonshot AI lists “automatic context caching” among the features of Kimi K3.1 Pro, according to our report on the model. The idea is now standard across major AI APIs.

The same briefing, sent again and again

Picture a coding assistant. Every time you ask a question, the app sends the model a long briefing: the rules it must follow, descriptions of its tools, and maybe a large chunk of your codebase. Only the last line, your new question, is different.

Without caching, the model reads the whole briefing from scratch each time, and you pay for it each time. With caching, the briefing is read once and kept. Later requests pay a small fee to reuse it and process only what is new.

Saving the model’s notes instead of the text

Before a language model can answer, it reads the prompt and builds internal notes for each token (a token is a small piece of text, roughly a word fragment). These notes are called the key-value cache, or KV cache. They are what the model looks back at while it writes its reply.

Normally those notes are thrown away after the reply. Prompt caching keeps them for a while instead. If a new request begins with exactly the same text, the server loads the saved notes and continues from there.

The key word is begins. Each token’s notes depend on everything before it, so only an identical opening can be reused. Anthropic’s documentation says a hit requires 100% identical prompt segments, including text and images, up to the point being cached.

Anthropic and OpenAI both discount reused text, but on different terms

The details differ by company, and they change often, so check the current documentation before budgeting.

Anthropic’s docs describe a default cache lifetime of 5 minutes, refreshed at no charge each time the cached content is used. A 1-hour option exists too. Writing to the cache costs more than normal input (1.25 times the base price for the 5-minute lifetime, 2 times for the 1-hour one). Reading from it costs 0.1 times the base price on most models, and less on some newer ones. Developers can mark cache points themselves, up to four per request, or let automatic caching move the point as a conversation grows.

OpenAI’s docs say caching is on by default for supported models. For GPT-5.6 and later, the docs list a 1,024-token minimum, a 30-minute default lifetime, and cache reads at 0.1 times the normal input rate (0.05 times on GPT-6.1 Sol). Earlier models have different rules.

Short prompts may not be cached at all. Anthropic’s minimum ranges from 512 to 4,096 tokens depending on the model, and a request below it is processed without caching and without an error.

Three 2023 papers turned cache reuse into working systems

Several research groups published the building blocks in 2023. A team led by Woosuk Kwon introduced PagedAttention and the vLLM server in September 2023. They treated the KV cache like operating-system memory, split into pages, so that cache could be shared inside and across requests with little waste.

In November 2023, In Gim and colleagues proposed Prompt Cache, which stores the saved notes of reusable text segments such as system messages and templates. They reported time-to-first-token reductions of about 8 times on a GPU and 60 times on a CPU, with no change to the model.

In December 2023, Lianmin Zheng and colleagues published SGLang. Its RadixAttention component reuses the KV cache across calls, and the authors reported up to 6.4 times higher throughput on their tasks. These are the authors’ own measurements on their own benchmarks.

Kimi’s 1M-token window shows why a reuse discount matters, and where pricing is heading

Large context windows make repeated text expensive. In our Kimi report, a request that fills the 1M-token window at the Vercel AI Gateway input price of $2 per million tokens costs about $2 before any output. Moonshot says cache hits are billed at a lower rate than misses. Resending the same long document across many calls is where that gap would add up.

Agents are the main beneficiaries. An agent often makes dozens of calls in a row, each repeating the same instructions and a growing history. Caching turns that repetition from a cost multiplier into a small line item.

Pricing is shifting. Both vendors now charge explicitly for cache writes and offer longer lifetimes (up to 12 hours on newer OpenAI models), which turns caching into something developers plan around rather than a hidden discount. Expect the share of each request that hits the cache to matter more than the headline price per million tokens.

One changed line near the top can silently switch the cache off

Because matching starts at the first token, one change near the top can break everything after it. Anthropic’s documentation gives an example: a breakpoint placed on a block that changes every request, such as a timestamp, never hits.

The same docs list other changes that invalidate cached content. Editing tool definitions invalidates the tools, system and message layers. Changing thinking settings or the output_config.effort setting invalidates the messages. For models with an effort setting, such as Kimi K3.1 Pro, that is worth knowing before switching it mid-conversation, though each provider’s rules differ.

Timing matters as well. Anthropic notes that the lifetime clock starts when the request begins, not when the response finishes, so a long streamed answer can use up much of a 5-minute window.

The practical habit is simple: put stable material first (instructions, tools, reference documents) and changing material last. Then check the usage fields. Anthropic reports cache_creation_input_tokens and cache_read_input_tokens; if both are 0, nothing was cached.