scaled-dot-attention — Twilio case

Twilio-inspired messaging model combines query, key, and value representations to select relevant context. You need to compute causal scaled dot-product attention: each position may attend only to itself and earlier positions, and the scores are scaled by the square root of the key dimension.

Compute causal scaled dot-product attention: scores $S=Q{K}^{\mathrm{\top }}\mathrm{/}\sqrt{d}S=QK^\backslash top/\backslash sqrt\; d$, set every score above the diagonal (future positions) to $-\mathrm{\infty }-\backslash infty$, take a row-wise softmax and return the weights times V. Row $ii$ therefore only attends to positions $0,\dots ,i0,\backslash dots,i$.

Implement solve(Q,K,V).

Returns. Return an array with one output row per query row. Q, K and V have the same number of rows.

Examples

Example 1

Input:

solve([[1.0, 0.0], [0.0, 1.0]], [[1.0, 0.0], [0.0, 1.0]], [[1.0, 2.0], [3.0, 4.0]])

Output:

[[1.0, 2.0], [2.339523, 3.339523]]

Example 2

Input:

solve([[1.0]], [[2.0]], [[5.0]])

Output:

[[5.0]]