Highway Networks: A Full Highway Layer
Intermediate
The problem, from first principles
A highway layer combines the two ideas: a nonlinear transform, and a learned gate that decides between transforming and carrying the input. Stacking many such layers trains as easily as a shallow network, which is the paper's main point.
From theory to code
Implement highway_layer(x, W_h, b_h, W_t, b_t), which computes H(x) with tanh, the gate T(x) with the sigmoid, and returns the blended output.
Constraints
W_handW_tare(D, D).- The output has the same shape as
x.
Hints
Hint 1
Build the transform and gate with the same affine maps as before, then reuse the blend formula.
Keywords
research-papersarchitecturehighway
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.