PagedAttention: KV Cache Size
Intermediate
The problem, from first principles
During generation a transformer keeps keys and values for every past token, in every layer. This KV cache is often the largest memory consumer at inference, and it sets how many requests a GPU can serve at once.
From theory to code
Implement kv_cache_bytes(layers, heads, d, seq, nbytes), the cache size for one sequence.
Constraints
- Keys and values are both stored.
Hints
Hint 1
Multiply two (keys and values) by the layers, heads, head dimension, sequence length and bytes per element.
Keywords
research-paperssystemsservingkv-cache
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.