KV cache
Per-token key/value tensors cached during generation so each new token doesn't recompute attention over the whole prompt. Its memory footprint — not compute — usually caps how many requests a GPU can serve.
Per-token key/value tensors cached during generation so each new token doesn't recompute attention over the whole prompt. Its memory footprint — not compute — usually caps how many requests a GPU can serve.