Starts Nov 23Live Cohort 3 of Vibe Coding & AI Agents: 3 weeks, 9 live hands-on sessionsCohort 3 (Nov 23) →
InfrastructureInsight · On RadarNVIDIA Developer Blog→ original

NVIDIA TensorRT Now Scales Generative AI Inference Across Multiple GPUs

NVIDIA has updated TensorRT: the engine now supports inference across several GPUs at once. Its key optimizations, including kernel fusion, memory management and quantization, are preserved when the workload is distributed. The update responds directly to the growth of generative models, which long ago stopped fitting into the memory of a single accelerator.

AI-processed from NVIDIA Developer Blog; edited by Hamidun News
NVIDIA TensorRT Now Scales Generative AI Inference Across Multiple GPUs
Source: NVIDIA Developer Blog. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceNVIDIA Developer Blog ↗
Editor-in-Chief
Kernel fusionfuses operations to minimize the overhead of transferring data between devices
Memory planningmanages VRAM across GPUs intelligently, without redundant tensor copying
INT8/FP8 quantizationapplied to the entire compute graph, not just to parts of it

NVIDIA has updated TensorRT with native support for inference across multiple GPUs at once. Large generative models can now run in production without manual sharding and without losing the engine's key optimizations.

Why one GPU is no longer enough

Modern generative models are growing faster than video memory. Diffusion networks for video generation, multimodal LLMs with extended context and complex media-content pipelines have long outgrown 80 GB, the ceiling of the flagship H100. Inference engineers faced a harsh choice: split the compute graph by hand and lose TensorRT's optimizations, or move to third-party frameworks with lower throughput.

TensorRT is the de facto standard for production deployment on NVIDIA hardware. The engine optimizes compute graphs at the kernel level: it fuses operations, plans memory use and applies quantization, which gives the lowest latency and highest throughput among the available options. The problem is that all of these optimizations previously worked only within a single GPU.

What multi-device inference offers

The new capability lets TensorRT automatically distribute a model across several GPUs while keeping the full set of optimizations:

  • Kernel fusion: fuses operations to minimize the overhead of transferring data between devices
  • Memory planning: manages VRAM across GPUs intelligently, without redundant tensor copying
  • INT8/FP8 quantization: applied to the entire compute graph, not just to parts of it
  • Tensor parallelism: automatically splits model weights across devices, with no manual code changes
  • Pipeline parallelism: different layers of the network run in parallel on different cards, increasing overall throughput

Previously, achieving a similar result required a complex combination of TensorRT with external tools, such as TensorRT-LLM or Triton Inference Server, and several weeks of engineering work. Now multi-device support is built into the engine itself.

Who needs this today

Teams building inference pipelines for media generation stand to gain the most: text-to-video systems, real-time content adaptation, interactive avatars and multimodal assistants. All of these tasks demand both a large model (meaning a lot of memory) and minimal latency (meaning no compromises on optimization).

The feature also changes the economics of inference in the cloud. Instead of wrestling with weight sharding across a GPU cluster and maintaining home-grown synchronization logic, teams can use the standard TensorRT API and get the same performance with lower development and maintenance costs.

The mid-size segment deserves a separate mention: companies that have two to four GPUs but no dedicated ML infrastructure team. For them, removing the barrier to multi-device inference is the biggest practical shift.

What this means

Scaling AI inference across multiple devices is moving from a "task for narrow specialists" to a "built-in engine feature." When TensorRT takes over distribution, the gap between a trained model and a scalable production service shrinks significantly. That directly affects which AI products mid-size teams can afford to launch.

Story

Key Questions Answered

Why one GPU is no longer enough?

Modern generative models are growing faster than video memory. Diffusion networks for video generation, multimodal LLMs with extended context and complex media-content pipelines have long outgrown 80 GB, the ceiling of the flagship H100. Inference engineers faced a harsh choice: split the compute graph by hand and lose TensorRT's optimizations, or move to third-party frameworks with lower throughput. TensorRT is the de facto standard for production deployment on NVIDIA hardware.

What multi-device inference offers?

The new capability lets TensorRT automatically distribute a model across several GPUs while keeping the full set of optimizations: Kernel fusion: fuses operations to minimize the overhead of transferring data between devices Memory planning: manages VRAM across GPUs intelligently, without redundant tensor copying INT8/FP8 quantization: applied to the entire compute graph, not just to parts of it Tensor parallelism: automatically splits model weights across devices, with no manual code changes…

Who needs this today?

Teams building inference pipelines for media generation stand to gain the most: text-to-video systems, real-time content adaptation, interactive avatars and multimodal assistants. All of these tasks demand both a large model (meaning a lot of memory) and minimal latency (meaning no compromises on optimization). The feature also changes the economics of inference in the cloud.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…