THE TRANSFORMER RESIDUAL CHECK

Difficulty: Hard Tags: Transformers, Residual Connections, Layer Normalization

Story

A model's pre-norm transformer block is ported to a custom inference runtime. The port accidentally normalizes after the residual addition instead of before the attention and feed-forward sublayers. Compute a simplified scalar-vector pre-norm block with all parameters supplied.

The Math

Every vector here has length 2. Layer normalization uses population variance over the vector's two elements, with learned scale gamma, bias beta, and epsilon:

LN(v) = gamma * (v - mean(v)) / sqrt(var(v) + eps) + beta

(mean and var are elementwise-independent scalars shared by both components; gamma, beta are 2-vectors applied elementwise.)

Attention and FFN are affine maps a(v) = A v + c and f(v) = F v + d (2x2 matrix, row-major). The pre-norm block computes:

u = x + a(LN_1(x))
y = u + f(LN_2(u))

Input Format

x (2-vector); gamma1, beta1, eps1 (2-vectors, 2-vectors, float); A (2x2, row-major), c (2-vector); gamma2, beta2, eps2; F (2x2, row-major), d (2-vector).

Output Format

y: the final 2-vector (6 decimal places).

Constraints

• All entries have absolute value at most 10; epsilons are positive.
• Time limit: 1.0 second.

Example

Input

x = [1, 3]
gamma1 = [1, 1], beta1 = [0, 0], eps1 = 1e-6
A = [[1, 0], [0, 1]], c = [0, 0]
gamma2 = [1, 1], beta2 = [0, 0], eps2 = 1e-6
F = [[1, 0], [0, 1]], d = [0, 0]

Output

[-0.999999, 4.999999]

Explanation: The first normalization maps (1,3) close to (-1,1). The residual gives a vector close to (0,4); the second normalization is close to (-1,1), and the final residual is close to (-1,5).