Highway Networks: The Transform Gate
Intermediate
The problem, from first principles
Very deep plain networks are hard to train because gradients vanish through many nonlinear layers. Highway networks (Srivastava, Greff & Schmidhuber, 2015) let each layer learn how much of its input to carry forward unchanged, using a learned gate.
From theory to code
Implement highway_gate(x, W, b), which returns the transform gate t = sigmoid(x W^T + b) for each feature.
Constraints
- Gate values are in
(0, 1). Wis square with shape(D, D).
Hints
Hint 1
Compute z = x @ W.T + b and apply the logistic function element-wise.
Keywords
research-papersarchitecturehighway
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.