Bahdanau Attention: Turning Scores Into Weights

The problem, from first principles

Scores can be any real numbers, but attention needs weights that are positive and sum to one, so the model can average the encoder states. A softmax does that: it exponentiates each score and normalizes.

From theory to code

Implement attention_weights(scores), a numerically stable softmax over the score vector.

Constraints

• Subtract the maximum before exponentiating to avoid overflow.

Hints