AI gateway: The checkpoint that controls how apps and agents reach AI models

An AI gateway is a checkpoint that sits between software and the AI models it calls. Every request and every answer passes through it, so one place can route traffic, count usage, inspect content and block what a company does not allow.

The idea has moved from developer tooling to security. CrowdStrike’s AI Kill Switch, covered in our news story, works at this layer first.

One front door instead of a dozen side doors

Picture a company where each team wires its own chatbot or agent straight to a model provider. Every team holds its own secret key. Nobody can say how much is spent, which data leaves the building, or who asked what.

An AI gateway replaces those side doors with one front door. Apps send requests to the gateway, and the gateway forwards them to the provider. The provider keys stay inside the gateway, and each app gets its own limited credential instead.

Vendors use the names AI gateway and LLM gateway loosely. Gravitee’s documentation describes the LLM gateway as the model-focused subset, with the wider AI gateway also covering tool calls and agent-to-agent traffic.

What the checkpoint actually does to each request

Vendor documentation from Cloudflare, LiteLLM and Gravitee lists a similar set of jobs. Each one is a vendor description of its own product.

• Routing: pick a provider and model by cost, speed or capability. LiteLLM presents 100+ providers behind one OpenAI-compatible endpoint, so existing apps need few code changes.
• Access and budgets: LiteLLM issues virtual keys per user, team or project, each with allowed models, budgets and rate limits.
• Fallbacks: if one provider has an outage, retry or switch to another. Gravitee also lists circuit breakers, which stop sending traffic to a failing service.
• Caching: Cloudflare’s AI Gateway can answer repeated requests from its cache instead of calling the model again, which saves time and cost.
• Logging: record requests, errors, tokens and cost. Tokens, the chunks of text models bill by, are the usual cost unit, so limits are often set in tokens rather than requests.
• Guardrails: check prompts and answers for sensitive data or attacks such as prompt injection, then block or redact.

No single gateway offers all of this on day one, and the feature list varies a lot between products.

Guardrails and the human-sized hole in them

The security value comes from the guardrails. A gateway can inspect a prompt before it reaches the model and the reply before it reaches the user. Palo Alto Networks documents this pattern in its Prisma AIRS gateway: checks run as request-side and response-side hooks, and they look for prompt injection, sensitive data, harmful content and off-topic use.

There are limits. A vendor blog notes that streamed answers, which appear word by word, cannot be fully checked before the first words are shown. Each check adds delay and, in some products, cost. Gravitee also says a plain HTTP proxy cannot judge tool calls made through newer protocols such as MCP (Model Context Protocol) or A2A (agent-to-agent), because it cannot read them. And a gateway’s guardrails do not replace the safety layer of the model provider.

From cost dashboard to agent control point

Early gateways were about money and reliability: one endpoint, one bill, automatic failover. Open-source projects such as LiteLLM and hosted services such as Cloudflare’s AI Gateway made that easy to adopt.

The shift came with AI agents, programs that take actions on their own. An agent that can call tools around the clock raises a different question than a chatbot does: who can switch it off? A gateway is a natural answer, because every model call it makes passes through one place.

That is the pitch in CrowdStrike’s Falcon Guardian. In its September 2026 blog post, CrowdStrike described a native AI gateway as a centralized control point for traffic among applications, agents, models and services, with policy decisions informed by data about the user, agent, device and identity. At that time the company called the gateway upcoming. Our news story reports it as generally available and describes the kill switch as a circuit breaker: a person cuts the agent’s link to its models, and the agent’s code is not deleted.

The blind spot: What never touches the gateway

A gateway only governs traffic that goes through it. If an agent calls a model by another route, on a laptop or from a different network path, the checkpoint never sees it. That is why CrowdStrike says it will add enforcement on the endpoint later, and why it describes one control point as not enough.

Three more trade-offs matter for buyers:

• Single point of failure: if every call depends on the gateway, an outage or a bad policy affects every app at once.
• Added delay: every hop and every inspection costs time.
• Humans in the loop: a kill switch is only as fast as the detection and the person who decides to use it.

Claims about real-time termination are the vendor’s own and have not been independently tested.

What to watch as agents multiply

Three things are worth following. First, whether gateways learn to read agent protocols such as MCP and A2A, so they can approve or deny individual tool calls rather than whole connections. Second, whether gateway controls are paired with checks on the device where the agent runs. Third, whether common standards emerge: searching found no neutral specification for what an AI gateway must do, and most descriptions come from companies selling one.

For a team choosing one, the practical questions are simple. Which traffic does it actually see? Who can stop an agent, and how quickly? And what happens to everything else if the gateway itself fails?