Transformer Feed-Forward

Compute the position-wise feed-forward network of a Transformer block: $\max (0,x{W}_1+b_1)W_2+b_2\backslash max(0,\backslash ,xW\_1+b\_1)\backslash ,W\_2+b\_2$, applied independently to every row of x.

Implement solve(x,W1,b1,W2,b2).

Returns. Return an array with the same number of rows as x.

Examples

Example 1

Input:

solve([[1.0, 2.0]], [[1.0, 0.0], [0.0, 1.0]], [0.0, 0.0], [[1.0, 0.0], [0.0, 1.0]], [0.0, 0.0])

Output:

[[1.0, 2.0]]

Example 2

Input:

solve([[1.0, -2.0]], [[1.0], [1.0]], [0.0], [[3.0]], [1.0])

Output:

[[1.0]]