Concepts

Time to first token

Reference entryInference and serving
Reference entryTime to first token

Time to first token is how long a model takes to start its reply after receiving a request, a key measure of how responsive an AI app feels.

Also known as TTFT.

Where it sits