TensorRT-LLM is NVIDIA's open-source library for running large language models quickly and efficiently on NVIDIA GPUs.
In plain words
Serving a big model efficiently takes more than loading its weights. TensorRT-LLM supplies tuned GPU kernels for common operations, a runtime and a Python API. It supports batching that mixes new and in-progress requests, paged KV-cache memory, low-precision formats such as FP8 and FP4 on newer NVIDIA chips, and splitting one model across many GPUs or machines.
A quick example
Imagine a company serving an open-weight model on a rack of NVIDIA GPUs. It loads the model through TensorRT-LLM's Python API, turns on FP8 on its H100s to save memory, and spreads the model over several GPUs with tensor parallelism.
Where you'll meet it
- High-throughput LLM serving on NVIDIA hardware.
- Serving on data-centre GPUs such as the H100 and B200.
Related
Sources
- DocsTensorRT-LLM README, NVIDIA (GitHub) · read 29 Sept 2026
- DocsTensorRT-LLM overview, NVIDIA docs · read 29 Sept 2026
Checked by WhatIsAI against the sources above · 29 Sept 2026. Spotted an error? Report it.