Warmup Schedule

Compute a learning rate with linear warm-up followed by cosine decay. For steps t < warmup the rate is lr0 * (t + 1) / warmup. From t = warmup on, with $q=\min (t-\text{warmup},\max (1,T-\text{warmup}))q=\backslash min(t-\backslash text\{warmup\},\backslash ,\backslash max(1,T-\backslash text\{warmup\}))$, the rate is ${\eta }_{\min }+\frac{1}{2}({\eta }_0-{\eta }_{\min })(1+\cos (\pi q\mathrm{/}\max (1,T-\text{warmup})))\backslash eta\_\{\backslash min\}+\backslash tfrac12(\backslash eta\_0-\backslash eta\_\{\backslash min\})\backslash big(1+\backslash cos(\backslash pi\; q/\backslash max(1,T-\backslash text\{warmup\}))\backslash big)$.

Implement solve(lr0, min_lr, t, warmup, T).

Returns. Return the learning rate as a float. It reaches lr0 at t = warmup - 1 and again at t = warmup, then decays to min_lr at t = T.

Examples

Example 1

Input:

solve(1.0, 0.1, 0, 2, 6)

Output:

0.5

Example 2

Input:

solve(1.0, 0.1, 2, 2, 6)

Output:

1.0

Example 3

Input:

solve(1.0, 0.1, 6, 2, 6)

Output:

0.1