Latency (TTFT / total)
Time-to-first-token is how long before output starts appearing; total latency is the full generation. Streaming makes TTFT the number users actually feel.
Time-to-first-token is how long before output starts appearing; total latency is the full generation. Streaming makes TTFT the number users actually feel.