Speculative Decoding: The Residual Distribution
Advanced
The problem, from first principles
When a drafted token is rejected, the replacement is sampled from the residual distribution, the part of the target that the draft under-covered. Together with the acceptance rule this keeps the output exactly target-distributed.
From theory to code
Implement residual_distribution(p, q), the positive part of the difference, renormalized.
Constraints
- The result sums to one when any residual mass exists.
Hints
Hint 1
Take the element-wise maximum of p minus q and zero, then divide by its sum.
Keywords
research-paperssystemsinferencedecoding
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.