Adam: Updating the Two Running Moments
The problem, from first principles
Plain gradient descent uses one global learning rate for every parameter. Adam (Kingma & Ba, 2014) keeps two running averages per parameter instead: one of the gradients themselves (the direction to move) and one of the squared gradients (how big the steps have been, so each parameter can get its own effective step size). This question implements only the bookkeeping that keeps those two averages up to date, which is the first thing Adam does on every training step.
From theory to code
Implement adam_update_moments(m, v, grad, beta1, beta2). It takes the previous first and second moment and the current gradient, and returns the two new moments. The signature and docstring are already in the editor.
Constraints
mandvstart at zero for a fresh parameter.beta1andbeta2default to the paper's values,0.9and0.999.- Works on scalars and NumPy arrays of the same shape.
Hints
Hint 1
Each moment is a weighted blend of its old value and a new quantity. The first moment blends the old m with grad; the second blends the old v with grad**2.
Hint 2
Use beta as the weight on the old value and 1 - beta as the weight on the new one. That's what keeps each average stable, instead of growing with the number of steps.
Keywords
No results yet
Run your code to check it against the examples, or submit to run every test.