Additive Attention Score

Compute additive (Bahdanau-style) attention scores between one query and several keys: $s_j=v^{\mathrm{\top }}\tanh (W_{q}q+W_k{k}_j)s\_j=v^\backslash top\backslash tanh(W\_qq+W\_kk\_j)$. Here query is a vector, keys has one key per row, Wq and Wk project them to a common hidden size and v maps the hidden vector to a scalar.

Implement solve(query, keys, Wq, Wk, v).

Returns. Return a NumPy vector with one score per key. The scores are unnormalised; a softmax turns them into attention weights.

Examples

Example 1

Input:

solve([1.0, 0.0], [[0.0, 1.0], [1.0, 0.0]], [[1.0], [0.0]], [[0.0], [1.0]], [1.0])

Output:

[0.964028, 0.761594]

Example 2

Input:

solve([0.0], [[0.0]], [[1.0, 2.0]], [[1.0, 2.0]], [1.0, 1.0])

Output:

[0.0]