Menu
Concepts

Latency

3 min readbeginnerUpdated 28 Sept 2026
1 · In one line

Request latency is the elapsed time between sending an inference request and receiving its final response.

1 · What it is

Latency needs two boundaries. For a streaming model, time to first token measures from request send to the first streamed output. Request latency continues until the final response.

Queue duration isolates time spent waiting for scheduling. Inter-token latency records gaps between consecutive streamed outputs after streaming begins.

GenAI-Perf reports average, p90 and p99 latency statistics. Metric terminology is not standardized across benchmarking tools. MLPerf’s server scenario models online applications with random query arrival.

2 · Why it exists

Request latency ends when the final response is received.

Different endpointsTime to first token ends when the first response is received.
Queueing mattersA request can spend time waiting in a scheduling queue.
Tails matterServing tools report percentiles such as p90 and p99 as well as averages.
3 · How it works

Mark one request from send time through its first and final responses.

TTFT ends at the first streamed output; request latency ends at the final response.
  1. 1 · startRecord when the benchmark sends the request.
  2. 2 · queueMeasure time spent waiting for the scheduler.
  3. 3 · firstStop the TTFT interval when the first response arrives.
  4. 4 · streamMeasure gaps between consecutive streamed outputs.
  5. 5 · finishStop request latency when the final response arrives.

Metric terminology is not standardized across benchmarking tools.

4 · Where it's used
WhoWhat they askWhat it works with
Product engineer“How long before a user sees the first token?”Time to first token
Serving operator“Are requests waiting before model execution?”Queue duration
Reliability engineer“How slow are the worst successful requests?”Tail-latency percentiles
5 · What it solves, and what it doesn't
solves
  • Request latency provides one value per request in a benchmark.
  • TTFT measures from request send to the first streamed output.
  • Queue duration measures time spent waiting for scheduling.
doesn't solve
  • GenAI-Perf reports averages and percentile latency statistics.
  • Large batch sizes can add a latency penalty for the first requests.
  • Metric terminology is not standardized across benchmarking tools.
  • MLPerf's server scenario models online applications with random query arrival.
6 · Go deeper
ThroughputTokens per secondContinuous batchingPrefill and decode

Sources used

This explainer is written in original language. The links below support its factual claims.

  1. docsGenAI-Perf metrics, NVIDIA · read 28 Sept 2026
  2. docsTriton Inference Server metrics, NVIDIA · read 28 Sept 2026
  3. docsvLLM benchmark CLI, vLLM · read 28 Sept 2026
  4. docsDynamic Request Batching, Ray · read 28 Sept 2026
  5. officialLlama 2 70B - An MLPerf Inference Benchmark for Large Language Models, MLCommons · read 28 Sept 2026
Suggest a correction