Free Intensive Nov 21–22Free 2-day hands-on Vibecoding Intensive on Nov 21–22 at 16:00 MSK (Sat–Sun) · Live Cohort 3 starts Nov 23Join Free →Cohort 3 (Nov 23)
ReleaseCode / SDK · ActionablePyTorch Blog→ original

Google and Meta Bring PyTorch to TPUs: 5,000 Hugging Face Models Run Out of the Box

Google and Meta have released TorchTPU, an open-source backend that runs regular PyTorch code on Google's AI chips (TPUs), not just on graphics cards. Previously, this required JAX or the older PyTorch/XLA, with code changes and long compilation times. In the simple case, switching the device from cuda to tpu is now enough, and the 5,000 most-downloaded Hugging Face models work out of the box. Trial migrations from GPU to TPU have become noticeably cheaper.

AI-processed from PyTorch Blog; edited by Hamidun News
Google and Meta Bring PyTorch to TPUs: 5,000 Hugging Face Models Run Out of the Box
Source: PyTorch Blog. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourcePyTorch Blog ↗
Editor-in-Chief
Key metric #1About 450 Aten operations, including the forward and backward passes
Key metric #2Already in production at some of Google Cloud's largest customers
Key metric #3Peak speed cannot be reached with a single swap: shapes and sharding are tuned for GPUs

Most AI teams write their models in PyTorch and run them on graphics cards, most often from Nvidia. Google has its own AI chips, Tensor Processing Units (TPUs), which are more economical in energy and cost, but familiar code did not simply run on them: you needed JAX or TensorFlow. Now Google, together with the PyTorch team at Meta, has released TorchTPU, an open-source backend, meaning a layer that translates PyTorch commands into the TPU language.

The project started in mid-2025. According to the authors, it already runs in production at some of Google Cloud's largest customers, for both inference (generating answers from a finished model) and training. Out of the box, it runs the 5,000 most-downloaded models from Hugging Face, the largest catalog of ready-made neural networks.

Why the Previous Attempt Didn't Catch On

The first version of PyTorch for TPUs, PyTorch/XLA, came out in 2020. According to its authors, it had four problems. Developers had to intervene in the source code and manually mark the sections to be compiled. Compilation significantly delayed startup. The familiar eager mode, where each operation executes immediately, was not supported. And new distributed computing techniques in PyTorch itself arrived there with a delay.

Imagine you have a working training script on a GPU. In the old scheme, you would have had to rewrite it for the peculiarities of TPUs. TorchTPU was built the other way around: eager mode takes priority, and existing programs should run with minimal changes.

What Changed in the Approach

The authors set a number of principles for themselves. The runtime has no dependency on JAX. Torch Dynamo is used for torch.compile, and performance should be close to JAX compilation. The distributed techniques DDP, FSDP, and DTensor are supported at the lowest level of PyTorch. Collective operations (data exchange between chips) are mapped to StableHLO operations rather than to a separate library like NCCL. Runtime and compiler errors must be reported with clear messages.

In practice, migration looks like this: instead of torch.device("cuda") you write torch.device("tpu"), and instead of the nccl backend you use tpu_dist. You can also avoid tying yourself to specific hardware and use torch.accelerator.current_accelerator(). If the TorchTPU package is installed on the machine, PyTorch will pick it up through the autoload mechanism.

The authors candidly warn that peak speed cannot be achieved with such a swap. Tensor shapes, sharding, and the set of operations in a model may be tuned for GPUs, but the model should run right away.

What Has Been Verified

Support for about 450 Aten operations is claimed, including the forward and backward passes for all the basic operations that mainstream architectures need. There are eager and compiled modes, including the new DeferAndFuse. Custom kernels can be written via JAX/Pallas and the Helion DSL. Compatibility with Transformers v5, Diffusers, and Kernels has been tested. Models that run out of the box include Qwen, DeepSeek, Gemma, Llama, Mistral, Whisper, CLIP, ViT, BERT, and T5. Sharding over DTensor, multi-host inference in vLLM and SGLang, training in TorchTitan, and orchestration via Ray have been confirmed.

Precision is described separately. The main format for matrix multiplications on TPUs is bfloat16 (a 16-bit number) with results accumulated in 32-bit format. TorchTPU enables it by default, and for the vast majority of tasks, the precision is sufficient. Higher-precision modes are possible, but they cost speed: roughly three times the compute for high and roughly six times for highest. They should only be enabled if you have found convergence problems in training.

Limitations

A TPU device can be accessed by only one user process. Therefore, TorchTPU uses a "one process per device" scheme, unlike JAX, where one process is assigned per host. Because of this, co-locating reinforcement learning (RL) training tasks does not yet work out of the box, and the team is looking for a workaround.

The second limitation: the XLA compiler expects tensor sizes to be known in advance. This is inconvenient for PyTorch, so "bounded dynamism" is being added. Maximum sizes are known at compile time, while exact sizes are determined at runtime. This avoids unnecessary recompilations when the shape changes.

What to do in practice: if you are choosing between GPUs and TPUs based on inference or training cost, an experiment is now cheaper. Run your model on a TPU with almost no changes and see where it hits limits with tensor shapes and sharding. That is where most of the optimization work will be.

Story

Key Questions Answered

Why the Previous Attempt Didn't Catch On?

The first version of PyTorch for TPUs, PyTorch/XLA, came out in 2020. According to its authors, it had four problems. Developers had to intervene in the source code and manually mark the sections to be compiled. Compilation significantly delayed startup. The familiar eager mode, where each operation executes immediately, was not supported. And new distributed computing techniques in PyTorch itself arrived there with a delay. Imagine you have a working training script on a GPU.

What Changed in the Approach?

The authors set a number of principles for themselves. The runtime has no dependency on JAX. Torch Dynamo is used for torch.compile, and performance should be close to JAX compilation. The distributed techniques DDP, FSDP, and DTensor are supported at the lowest level of PyTorch. Collective operations (data exchange between chips) are mapped to StableHLO operations rather than to a separate library like NCCL. Runtime and compiler errors must be reported with clear messages.

What Has Been Verified?

Support for about 450 Aten operations is claimed, including the forward and backward passes for all the basic operations that mainstream architectures need. There are eager and compiled modes, including the new DeferAndFuse. Custom kernels can be written via JAX/Pallas and the Helion DSL. Compatibility with Transformers v5, Diffusers, and Kernels has been tested. Models that run out of the box include Qwen, DeepSeek, Gemma, Llama, Mistral, Whisper, CLIP, ViT, BERT, and T5.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…