Google’s Chrysalis agent fixed a simulated outage in 1.8 seconds, with humans signing off

Google Cloud has published Project Chrysalis, a research reference architecture for AI agents that write, sign and mount their own fixes during a live outage. In simulated Black Friday failures, Google says time to first mitigation fell from 15-45 minutes to 1.8 seconds.

A simulated cache stampede, patched without a restart

The scenario is a flash-sale surge that empties the Memorystore (Redis) cache and floods Cloud SQL until its connection pool saturates. In Google’s simulation, the hit ratio drops from 100% to 0%, P99 latency reaches 312.8 to 519.8 ms, and HTTP 500 errors reach 72.7% to 80.0%. Google notes that human responders and naive language models tend to restart the database pool, which it says drives latency up nearly tenfold in this scenario.

Chrysalis works differently. A supervisor agent built with the Agent Development Kit (ADK) and powered by Gemini 4 Argon diagnoses pool exhaustion. It then writes its own middleware, an in-memory token-bucket rate limiter with 10-token capacity and 40 requests per second replenishment, and mounts it in the API Gateway. In a second simulated surge, Google reports 0.0% server errors, 0% pool saturation and P99 latency of 18.2 ms. Excess traffic is shed with HTTP 429: 40 requests throttled, 10 served from cache.

Afterward the system scores each action by its effect on telemetry. The database restart is penalized, while the rate limiter earns +0.97 credit. A signed rule saying restarts are contraindicated during cache stampedes goes into vector search. The agent then proposes a permanent Traffic Shaping Specialist Agent. Once an on-call engineer approves it, the agent graph is rewired in memory, which Google says needs no container restart.

Google also reports that a BigQuery replay of 400 counterfactual policies finished in about 0.5 ms. It found a better policy (5-token capacity, 5.0 requests per second). A chaos agent then purged the cache and sent 200 concurrent requests in a sandbox, and Google reports 0 server errors and 13% maximum database load.

Four gates stand between the agent and production

The design is mainly a governance argument. Google describes a chain every patch, memory and topology change must clear:

• Model Armor screens incoming alerts for prompt injection and sensitive data, and an AST sandbox checks generated code for forbidden modules and dynamic execution.
• Cloud KMS signs artifacts, and Binary Authorization refuses to mount anything whose signature does not verify.
• An Open Policy Agent controller enforces service-level floors, such as a replenishment rate of at least 10 requests per second, so the agent cannot throttle the service into an outage.
• Structural changes pause for dual-custody human approval, and every step is logged to Cloud Logging.

The research combines three ideas: counterfactual simulation (Dream-RSI), a three-layer evolution scheme (APEX) and Shapley-based credit assignment (AgentEvolver). Google has put the reference code on GitHub. It is a demo, not a managed product, and the post lists no pricing or regions.

Governance is the product, and the evidence is thin

The safety angle is the most useful part. An agent that writes code in production widens the attack surface, so Google pushes trust out of the model and into signatures, policy and people. Nvidia took a similar view when it moved agent security toward hardware. According to our report, Nvidia says its Sentry software on BlueField-4 chips can quarantine a misbehaving agent within milliseconds. The report notes Nvidia has published no independent tests of that claim. The two designs enforce limits at different layers, Google’s in cloud policy and Nvidia’s in silicon. Neither has independent validation yet.

The same caution applies here. Every number comes from Google’s own demo, run on a single failure type with a service built for the purpose. The headline ranges, 15-45 minutes and 72.7-80.0%, are the demo’s baselines, not measurements from real incidents. The post says nothing about how often generated patches fail the gates, how many false rejections the checks produce, or how the agent behaves on failures it has not rehearsed. The 1.8 seconds also leaves out the human approval step, which Google keeps for permanent changes.

It also helps to separate what is autonomous from what is not. The hotpatch and the memory write run on their own, while the rewiring of the agent hierarchy waits for a person. The design therefore keeps humans on structural decisions, but not on the emergency fix itself. Teams that adopt it would need to decide whether that line suits their own risk tolerance.

For readers who run production services, the practical value is a worked pattern rather than a tool. The pattern covers signing generated code, setting policy minimums the agent cannot cross, and logging every decision. Teams on Google Cloud can read the code and adapt it. Anyone expecting a drop-in on-call robot will not find one in this post.