Inside Google's AI Infrastructure Plan to Run AI Agents Around the Clock

November 12 2026 02:21
Google held the keynote on November 9 at 9 AM PT from its Silicon Valley campus, standing outside its first shipping container data center. Amin Vahdat, who leads AI and infrastructure at Google, framed the whole talk around one shift: a search query touches thousands of servers, returns an answer and is done, while an agent gets a goal, breaks it apart, writes code, calls tools, checks its own work and keeps looping.



That loop is expensive. Vahdat said a human chat might use about 1,000 tokens, while an agent doing the same job can fire hundreds of model queries and burn millions of tokens. Google says agent traffic grew by more than 7,800% over the past year, and that inference now makes up 50% to 70% of AI compute use, up from 40% two years ago. It also cited industry forecasts that put global token demand at more than 120 quadrillion a month by 2030, which it says implies a 55 times jump in compute intensity.


Google had already split its TPU line in two at Cloud Next in April, with one chip for training and one for inference. The keynote added two more systems to that eighth generation, and the keynote transcript is a bit garbled on their names, so I will describe them by what they do.

1. The first is built for confidential inference. It shares its silicon base with the training chip, with 12.6 petaflops of FP4 compute and 216GB of high bandwidth memory. Working with custom AMD hosts, it adds encrypted PCIe links, cryptographic attestation and secure memory enclaves, so a company can run proprietary models without trusting the software layer alone.
2. The second is built for huge contexts. It doubles the silicon per package, with 384GB of memory, 17.2TB per second of memory bandwidth and 768MB of on chip SRAM. Vahdat compared it to widening a highway: the cars do not go faster, but more of them fit, meaning more agents can run at once.

Both sit next to the chips Google described earlier. The training chip scales to a 9,600 chip superpod. The inference chip replaced the usual 3D torus network with a design called Boardfly, which cuts the worst case path from 16 hops to seven, and Google claims an 80% gain in performance per dollar for it over the previous generation.






Hardware only matters if people can use it, and most AI developers write PyTorch. Mark Lohmeyer, who runs AI and computing infrastructure at Google Cloud, spent his half of the keynote on that gap. TorchTPU, which Google open sourced on October 20 after building it with the PyTorch team at Meta, is meant to let existing PyTorch code run on TPUs with few changes.



Google also said vLLM and SGLang now have native TPU support, and it announced general availability of Nvidia Vera Rubin machines on Google Cloud. The message is that you should not have to pick a side: TPUs and GPUs both get first class treatment.


Agents spend a lot of time waiting on a tool call or a human reply, and an idle accelerator is wasted money. Google announced TPU Sync, which moves an agent's working memory off the chip and onto server memory while it pauses, then streams it back when work resumes.




The other half of the answer is routing. Agent prompts are long and repetitive, so sending each request to a random chip forces it to reread the same context. The llm-d project, which Google helped found, reached version 1.0 with backing from Red Hat, IBM, CoreWeave and Nvidia. It sends requests to chips that already hold the right cache and keeps prompt reading and token generation on separate pools.



The 1.0 release also adds multitenant quality of service, so background agent jobs do not starve interactive users. Mistral's engineers said flow control lets them hold or shed lower priority requests when capacity runs out, which protects latency for paying customers. Google is bringing llm-d into its GKE Inference Gateway.


Lohmeyer's line was that CPUs are making a comeback, because a model that thinks still needs somewhere to act. Google described three needs for that: strong isolation, instant wake up, and state that survives pauses. Its answer is Agent Compute Engine, which runs each agent in its own microVM, pauses it when it goes quiet, and restores memory and disk state when work resumes. Google says this lets it pack thousands of microVMs onto one host. Credentials stay outside the sandbox, and the Titanium chips handle networking and encryption on separate silicon.



For teams that want to run this themselves, there is Agent Substrate, an open source runtime that works with standard Kubernetes APIs. Google is donating Agent Substrate and gVisor to the Cloud Native Computing Foundation, along with OpenRL, and it compared the move to what it did with Kubernetes ten years ago. Ant Group, Modal, and Tines are joining as maintainers. One caveat: at least one outlet described the donation as a plan rather than a finished transfer, so check the project's status before you build on it.



Google also pointed to its governance tools: an agent gateway, a registry of approved tools, agent identity, observability and token budgets. On the CPU side, it said its Axion Arm instances offer up to 30% better price performance than the next leading hyperscaler.


The keynote also included a short update from Project Suncatcher, Google's research into running machine learning in orbit on solar power. Google launched its prototype satellite on October 1 and has since run early tests on the TPUs, including a hello world moment where a prompt went up, ran on the chips and came back for checking on the ground. A pair of satellites is planned for next year to test laser links between them.

This is a long bet, and the hard parts remain. NPR reported that radiators are among the heaviest parts of the current satellite, which matters because weight raises launch costs, and that rocket fuel raises its own environmental questions.

The most interesting part of the keynote was not a chip. It was the repeated insistence that codesign should not mean closed. Google is selling TPUs, but it spent much of its time on PyTorch, vLLM, Kubernetes and a foundation handoff, which suggests it knows developers will not rewrite their stack for one vendor.