cosine-learning-rate — Adobe case

Adobe-inspired model-training pipeline changes its learning rate over the course of a training schedule. You need to calculate the cosine-decayed learning rate at a requested training step so the scheduler matches the experiment configuration.

Compute the cosine-annealed learning rate at step t of T: ${\eta }_{\min }+\frac{1}{2}({\eta }_{\max }-{\eta }_{\min })(1+\cos (\pi t\mathrm{/}T))\backslash eta\_\{\backslash min\}+\backslash tfrac12(\backslash eta\_\{\backslash max\}-\backslash eta\_\{\backslash min\})\backslash big(1+\backslash cos(\backslash pi\; t/T)\backslash big)$. Note the argument order (t, T, lr_max, lr_min). t is not clamped, so for $t>Tt>T$ the cosine starts rising again.

Implement solve(t,T,lr_max,lr_min).

Returns. Return the learning rate as a Python float. T must be positive.

Examples

Example 1

Input:

solve(5, 10, 0.1, 0.0)

Output:

0.05

Example 2

Input:

solve(0, 10, 0.1, 0.01)

Output:

0.1

Example 3

Input:

solve(10, 10, 0.1, 0.01)

Output:

0.01