How to use it
- Choose a schedule.
- Enter the initial learning rate, the step t you want to check and the total T.
- Optionally set warmup steps, and the schedule’s own settings: decay interval and γ, final rate or power.
- Read the learning rate at t, its share of the initial rate and whether t is in warmup.
Formulas
With warmup W, τ = t − W counts steps after warmup and L = T − W is the decay length.
- Warmup (t < W): lr = lr₀ × t ÷ W
- Step: lr = lr₀ × γ^⌊τ ÷ s⌋
- Exponential: lr = lr₀ × γ^τ
- Cosine: lr = min + (lr₀ − min) × (1 + cos(π τ ÷ L)) ÷ 2
- Linear: lr = lr₀ + (min − lr₀) × τ ÷ L
- Polynomial: lr = (lr₀ − min) × (1 − τ ÷ L)^p + min
For cosine, linear and polynomial, τ stops at L: after T steps the rate stays at the final value.
Worked example
Cosine annealing from 0.001 to 0 over 100 epochs, no warmup, at epoch 25:
- τ ÷ L = 25 ÷ 100 = 0.25, cos(π × 0.25) = 0.70711;
- (1 + 0.70711) ÷ 2 = 0.853553;
- lr = 0.001 × 0.853553 = 0.0008535534, that is 0.853553 of the initial rate.
With 10 warmup epochs, epoch 5 gives 0.001 × 5 ÷ 10 = 0.0005; epoch 55 is 45 of the 90 decay epochs — exactly halfway — and also gives 0.0005.
Step decay from 0.1, ×0.1 every 30 epochs: at epoch 65, ⌊65 ÷ 30⌋ = 2 decays, so lr = 0.1 × 0.1² = 0.001.
Tips
- Warm up large models. A few hundred to a few thousand warmup steps keep Adam from taking huge first steps while its statistics are still noisy.
- Keep a floor. A small final rate (for example 10 % of the initial one) often works better than decaying all the way to 0.
- Count the total steps exactly: epochs × batches per epoch. The train/test split calculator gives the batches per epoch.