Google doubles TPU memory and compute in a single rack

When an agent works through a large repository, an accelerator may not have enough memory for the model and its working context. And when processing private documents, a different question arises: who gets access to the data during computation? Google is offering two new TPU configurations for these tasks. For a team choosing infrastructure, this is a reason to add one more option to a comparison test.

According to The Decoder, Google has expanded its eighth generation of accelerators with the TPU 8i Max and TPU 8tx systems. Together with the earlier variants, the company counts four TPU systems in a year. It is important to distinguish between the processor and the system built around it: both new products use existing chips. What changes is their layout, memory and network connections.

Two chips help fit more work

In the TPU 8i Max, Google placed two TPU 8i chips in a single package. The company claims double the compute, memory capacity and memory bandwidth while keeping the rack format. For a data center, this is a promise to fit more computing resources into the same space.

Imagine a service where several agents simultaneously study code, read documentation and propose fixes. Besides the model weights, the infrastructure has to store intermediate data from processing requests. The longer the context and the more concurrent users, the heavier the load on memory. Extra capacity may allow the service to handle heavier requests or increase the number of parallel tasks.

But doubling hardware resources does not by itself mean a twofold speedup of an application. The result depends on the model, request length, how computation is organized, and where exactly the delay occurs. If a service is waiting for a response from an external API, additional accelerator memory will not fix that part of the work.

The material provided contains no prices, independent measurements or detailed power consumption comparison. So the announcement does not let us deduce how much cheaper a request will become or how many users a specific product can sustain. These figures will have to be obtained through testing.

Protecting data during computation

TPU 8tx is aimed at companies and government organizations that work with confidential data. Encryption at rest and in transit covers only part of the problem: during processing, data usually becomes available to the computing environment in plaintext. According to Google, the new system protects and isolates data and model weights while they are in active use as well.

For example, an organization wants to fine-tune a model on internal documents and then derive a smaller model for everyday work. Google describes the ability to perform fine-tuning, distillation and reinforcement learning for agents in one system, without moving sensitive data between different clusters.

For such a scenario, the entire protection scheme matters. The brief description does not allow us to conclude that the system covers every threat or automatically meets a specific organization's requirements. Before choosing infrastructure, you need to study the isolation boundaries, key management and access conditions. Nvidia also offers similar confidential computing capabilities, so specific implementations will have to be compared.

Code portability will determine practical value

Hardware specifications help little if migrating an application requires rewriting a significant part of the code. According to the material, Google has open-sourced TorchTPU to simplify running PyTorch on TPUs; support for vLLM and SGLang is also stated. This lowers the potential barrier to entry, although the promise of minimal changes still needs to be verified against your own dependencies and model operations.

A practical next step is a small pilot with a real workload. For TPU 8i Max, it is worth measuring memory consumption, service throughput and latency under concurrent requests. For TPU 8tx, a check of confidentiality requirements is added. In both cases, what must be counted is the full cost of serving the workload, including migration and maintenance.

For a founder or product manager, the meaning of this news is a wider choice of infrastructure. If a product is already bumping into memory or data protection requirements, the new TPUs deserve a test. The decision to switch will come after measurements: the announcement of four configurations does not by itself prove the application's economics.