Triton Inference Server is NVIDIA's open-source server software for running trained AI models in production, so apps can send it requests and get predictions back.
In plain words
After training, a model has to be served: kept loaded and ready to answer requests. Triton loads models from a folder called a model repository, and it works with models from TensorRT, PyTorch, ONNX, OpenVINO and more. It can run several models at once and group incoming requests into batches, which keeps the hardware busy. It can use NVIDIA GPUs, ordinary x86 or ARM processors, or AWS Inferentia chips.
A quick example
Imagine a photo app with an image classifier and a text model. The team places both in one Triton model repository. When many users upload photos at the same moment, Triton can combine those requests into one batch for the GPU instead of handling them one by one.
Where you'll meet it
- Serving models in the cloud, in data centres and at the edge.
- Real-time, batched and streaming prediction workloads.
Related
Sources
- DocsNVIDIA Triton Inference Server, NVIDIA documentation · read 29 Sept 2026
- Docstriton-inference-server/server, NVIDIA (GitHub) · read 29 Sept 2026
Checked by WhatIsAI against the sources above · 29 Sept 2026. Spotted an error? Report it.