Compute TTFT, TPOT, ITL, and Token Throughput

The problem, from first principles

[01-latency-percentiles] measures the SPREAD of latency across requests, but LLM serving has its own vocabulary for what happens WITHIN a single streamed response. Given the timestamps at which each token of a generated response was produced (plus the request's arrival time), compute the four standard LLM serving metrics: time-to-first-token (TTFT), time-per-output-token (TPOT), inter-token latency (ITL) per gap, and overall token throughput.

From theory to code

TTFT = token_timestamps[0] - request_arrival_time
ITL_i = token_timestamps[i] - token_timestamps[i-1]     for i = 1..n-1
TPOT = mean(ITL)
throughput = n_tokens / (token_timestamps[-1] - request_arrival_time)

Constraints

• TTFT is measured from request_arrival_time, not from the first token's own generation start.
• ITL is the list of gaps between consecutive token timestamps (length n_tokens - 1).
• TPOT is the mean of the ITL list (None if fewer than 2 tokens).
• Throughput is total tokens divided by total wall-clock time from arrival to the last token.

Hints