Multi-Query Attention: The Cache Saving Ratio
Beginner
The problem, from first principles
The benefit of multi-query attention is a ratio: each key-value head removed from the cache saves its share of memory and bandwidth. The ratio of query heads to key-value heads is that saving.
From theory to code
Implement saving_ratio(H, kv_heads), the cache reduction factor.
Constraints
- Returns a float.
Hints
Hint 1
Divide the query head count by the key-value head count.
Keywords
research-paperssystemsattentioninference
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.