Learning-rate schedules

A learning-rate schedule changes how big each training step is over time, often starting with a short warmup and then lowering the rate.

Also known as warmup, cosine decay.