Adam First Step

Perform one bias-corrected Adam update. Given weights w, gradient g, first and second moment estimates m, v and the step number t (starting at 1), update the moments with beta1 and beta2, correct their initialisation bias, and move the weights by lr * m_hat / (sqrt(v_hat) + eps).

Implement solve(w, g, m, v, t, lr=0.001, beta1=0.9, beta2=0.999, eps=1e-08).

Returns. Return a tuple (new_w, new_m, new_v). Defaults: lr=0.001, beta1=0.9, beta2=0.999, eps=1e-8.

Examples

Example 1

Input:

solve([1.0], [1.0], [0.0], [0.0], 1)

Output:

([0.999], [0.1], [0.001])

Example 2

Input:

solve([2.0], [0.0], [0.0], [0.0], 1)

Output:

([2.0], [0.0], [0.0])