Sliding-window / local attention: bounding context to a fixed window

The problem, from first principles

[03-attention-quadratic-complexity] showed attention's compute and memory both scale QUADRATICALLY with sequence length. [04-flash-attention] removed the quadratic MEMORY cost without touching the quadratic COMPUTE cost (it computes the exact same full-attention answer, just without materializing the whole matrix at once). Sliding-window (or "local") attention takes a genuinely different approach: it reduces the COMPUTE itself, by simply refusing to let any query attend to keys beyond a fixed distance away. Instead of [04-seq-modeling/04-attention/02-causal-mask]'s causal mask (attend to EVERYTHING up to and including the current position, an unbounded, growing window as the sequence gets longer), sliding-window attention bounds every position to a FIXED-size window of window_size most-recent positions, regardless of how long the overall sequence is.

This turns attention's cost from O(seq_len^2) back into O(seq_len * window_size), LINEAR in sequence length (since window_size is a small, fixed constant, not something that grows with seq_len), a real, direct reduction in the amount of work, at the cost of every position genuinely losing access to anything outside its window (a real modeling tradeoff, not merely an implementation detail: information from far in the past can only reach later positions by being relayed forward, block by block, through intermediate positions still within range).

From theory to code

Implement build_sliding_window_mask(seq_len, window_size), an additive mask (like [02-causal-mask]'s) that permits position i to attend ONLY to positions j with i - window_size < j <= i, and sliding_window_attention(query, key, value, window_size), [01-scaled-dot-product-attention]'s attention with that mask applied.

Constraints

• Sliding-window masking is STILL causal: no position ever attends to the future (j > i is always forbidden), exactly like [02-causal-mask].
• Additionally, positions too far in the PAST are forbidden: j <= i - window_size is also masked out (-inf).
• Every position attends to AT MOST window_size positions total (itself plus up to window_size - 1 earlier ones); positions near the very start of the sequence, with fewer than window_size - 1 earlier positions available, simply attend to however many exist.
• window_size == seq_len must reduce EXACTLY to [02-causal-mask]'s ordinary (unbounded) causal mask.

Hints

positions = np.arange(seq_len)
distance = positions[:, None] - positions[None, :]
allowed = (distance >= 0) & (distance < window_size)
mask = np.where(allowed, 0.0, -np.inf)