AdamW Decoupled Decay

Perform one AdamW update: the Adam step of the previous problem plus decoupled weight decay, $w\leftarrow w-\eta (\widehat{m}\mathrm{/}(\sqrt{\widehat{v}}+\epsilon )+\lambda w)w\backslash leftarrow\; w-\backslash eta\backslash big(\backslash hat\; m/(\backslash sqrt\{\backslash hat\; v\}+\backslash varepsilon)+\backslash lambda\; w\backslash big)$, where wd is $\lambda \backslash lambda$ (default $0.010.01$).

Implement solve(w, g, m, v, t, lr=0.001, beta1=0.9, beta2=0.999, eps=1e-08, wd=0.01).

Returns. Return a tuple (new_w, new_m, new_v). Defaults: lr=0.001, beta1=0.9, beta2=0.999, eps=1e-8, wd=0.01.

Examples

Example 1

Input:

solve([2.0], [1.0], [0.0], [0.0], 1, lr=0.1, beta1=0.0, beta2=0.0, eps=0.0, wd=0.1)

Output:

([1.88], [1.0], [1.0])

Example 2

Input:

solve([1.0], [0.0], [0.0], [0.0], 1, lr=0.1, beta1=0.0, beta2=0.0, wd=0.1)

Output:

([0.99], [0.0], [0.0])