Hot AI News

Tool retrieval for AI agents: How deferred loading keeps hundreds of tools affordable

Tool retrieval lets an AI agent see only a short menu of tools and load full instructions for the one it needs. Here is how it works, where it came from and where it falls short.

Tool retrieval for AI agents: How deferred loading keeps hundreds of tools affordable
Photo illustration: Aiholics

Tool retrieval is a way for an AI agent to carry many tools without reading about all of them every time. The model sees a short menu of tool names. When a task needs one, the full instructions for that tool are loaded on demand.

It is also called deferred tool loading or tool search. The goal is to cut the number of tokens, the small text units models read and bill by, spent on tool descriptions.

A restaurant menu versus the whole recipe book

Picture a waiter who must memorize every recipe in the kitchen before taking an order. Most of that effort is wasted, because a table orders one or two dishes. A better waiter holds a one-line menu and fetches the full recipe only when the kitchen needs it.

An AI agent has the same problem. A tool, such as “issue a refund” or “create a pull request”, is described to the model with a name, a description and a detailed schema. The schema lists every parameter the tool accepts. Normally all of these go into the prompt on every request, whether or not they are used.

Advertisements

Why long tool lists cause trouble

A curved screen shows a violet AI task card surrounded by a dense field of abstract tool cards.
Photo illustration: Aiholics

Each schema costs tokens, and they add up. Anthropic’s engineering team wrote in November 2025 that a five-server setup (GitHub, Slack, Sentry, Grafana and Splunk) came to 58 tools and about 55,000 tokens before the conversation began. It said it had seen tool definitions use 134,000 tokens before optimization.

Cost is only part of it. More tokens mean more latency. Anthropic’s documentation also says selection accuracy tends to degrade once an agent has more than roughly 30 to 50 tools available. Google makes a similar argument for its own feature: a growing candidate list lowers tool-selection accuracy.

The steps of a deferred tool call

The pattern is similar across vendors. The developer marks most tools as deferred and adds a search tool. The model first sees only the search tool, any tools left loaded, and a name and short description for each held-back tool.

  1. The model decides it needs a capability and searches for it.
  2. The system returns the matching tools and expands their full definitions into the conversation.
  3. The model calls the tool as usual.

In Gemini’s version, according to Google, the developer adds a tool_search tool and sets defer_loading: true on the tools to hold back. The API fetches the definition and the function call happens within a single request. Our report on the Gemini launch covers the details, including that Google lists no extra fee beyond ordinary per-token prices.

The search itself can work in different ways. Anthropic’s tool search comes in a regex variant, where the model writes a text pattern, and a BM25 variant, where it writes a natural-language query. BM25 is a classic keyword-ranking formula from search engines. Anthropic’s docs also allow a custom search, such as one built on embeddings (numeric fingerprints of meaning). Google offers a server-side default and a client-side option declared with "execution": "client", which adds one round trip per search.

Advertisements

Where the idea came from

Retrieving tools is older than the current products. Researchers have long noted that a model with limited context cannot hold a huge API catalog. In 2025, Zhengliang Shi and colleagues published ToolRet at ACL 2025 (arXiv, March 2025), a benchmark with 7.6 thousand retrieval tasks and a corpus of 43 thousand tools. Their finding was that retrieval models that do well on conventional search benchmarks perform poorly at finding tools. They also reported that weak retrieval lowers how often tool-using models complete tasks.

The idea reached production APIs with Anthropic’s Tool Search Tool, announced on November 24, 2025. Its documentation describes a flag, defer_loading, that keeps a tool out of the model’s context until a search finds it. Google’s Gemini Interactions API now offers a version, in preview, for Gemini 3.6 and later models, according to Google.

What the published numbers show, and what they do not

Anthropic’s example compared a traditional setup of about 77,000 tokens of context used before work began with about 8,700 tokens using tool search. It called that an 85% reduction. On its internal evaluations of tools served through the Model Context Protocol (MCP, a standard for connecting agents to tools), it reported Opus 4 rising from 49% to 74% accuracy and Opus 4.5 from 79.5% to 88.1%.

These are the vendor’s own results on its own setups, and the article does not reconcile every figure it gives. Treat them as an illustration of the size of the effect, not a guarantee. Savings depend on how many tools you have and how large their schemas are.

Deferral also works with MCP servers. Both Google and Anthropic describe a way to defer every tool a server exposes, so a large server can be connected without listing its tools one by one.

Trade-offs and what to watch

Tool retrieval adds a step. The model must search before it can act, so a task that would have taken one call can take two. Anthropic’s guidance is that it pays off with 10 or more tools, or with definitions over about 10,000 tokens. It is less useful with few tools, or when every tool is used on every request.

The model can also fail to find the right tool. It only sees a name and short description, so vague naming hurts. Anthropic advises descriptive keywords, namespaced names such as github_ and keeping the three to five most-used tools loaded up front. Google says each stage appears in interaction.steps as tool_search_call, tool_search_result and function_call, which helps developers see what was searched and loaded.

Caching is another detail. Both companies say deferral is designed to work with prompt caching, because deferred schemas stay out of the cached start of the prompt. Anthropic’s docs add a constraint: a deferred tool cannot carry a cache marker, and the full tool list must still be sent on each request. Anthropic’s docs list a limit of 10,000 deferred tools per request.

Three things are worth watching. One is whether search quality keeps up as catalogs grow, which is the gap the ToolRet authors highlighted. Another is whether more platforms adopt the same pattern. The third is whether developers move toward custom, permission-aware search over their own registries, which both Anthropic’s and Google’s designs allow.

Sign Up for the Daily AI Pulse

One email a day. All the stories that matter.

Daniel ReedAI Research, Safety & Ethics Analyst

Daniel Reed currently works as an AI Research, Safety & Ethics Analyst at Aiholics, writing about how changes in artificial intelligence are affecting and will affect scholarship,…

Leave a Reply

Your email address will not be published. Required fields are marked *

You may also like