Ch.03
Learning Rate Scheduling: Slow Down When Needed
Imagine hiking blindfolded from a mountain peak toward the valley floor. At first, long strides get you down quickly. But near the bottom, the same big steps can overshoot the minimum and send you zigzagging up the opposite slope.
In Ch.02, the optimizer acted like a compass — choosing which way to step. In this chapter, the learning rate scheduler is the brake and accelerator — deciding when to narrow your stride. We explore practical strategies: wide strides for early exploration, small precise steps near the goal, so the model reaches its full potential.
Ch.02 optimizer = compass (direction). Ch.03 scheduler = brake & gas (stride). Compare LR curves vs epoch below.
FixedStepCosineWarmup+Cosine
Cycles Fixed → Step → Cosine → Warmup+Cosine. The dot tracks how large the stride is now.
Stride rhythm: check error → take a step → set next stride
Each epoch/step, the scheduler sets the next stride for the optimizer
- ① Fixed LR — same stride throughout; may overshoot the minimum late
- ② Step decay — sharp cuts at set epochs, like floors 30/60/90
- ③ Cosine — smooth slowdown — common for long runs & text AI
- ④ Warmup+Cosine — warm up then glide down — pairs with Ch.02 optimizer
Learning Rate Scheduling: Slow Down When Needed
1. What is learning rate scheduling? (The fixed-stride trap)
Concept: Ch.02 optimizers pick direction; scheduling picks how large each stride () is over time. Each step, the optimizer receives a different , so the step size changes even in similar situations.
Analogy: Hike fast on wide trails early, then take tiny steps near the valley floor. One fixed the whole run is too slow early and overshoots the minimum late, causing endless oscillation.
Tip: After Ch.02 sets direction, Ch.03 is stride over time. Jot down ‘stride + error’ together in your notes so you can see when braking happened and how scores responded.
2. Step decay: stair-step braking
Concept: At fixed intervals (epochs), cut sharply — often ×0.1 (). Cut every N intervals, or at several checkpoints like 30, 60, 90.
Analogy: An elevator slowing at set floors. You only pick when to brake; the scheduler handles the rest.
Tip: Common for long training such as image classification. too small or too early → under-learning. Note how often and when you brake.
3. Cosine & Warmup: smooth slowdown and warm-up
Concept: Cosine shrinks smoothly for late fine-tuning. Warmup only at the start, ramping from 0 to target to avoid a huge early shock.
Analogy: Cosine = roller coaster easing to a stop; Warmup = revving a cold engine instead of flooring the gas.
Tip: For large models (text/image AI), warm-up + smooth decay is very common. Warmup is often 5–10% of total training; with Ch.02 optimizer it cuts early error spikes.
4. Plateau & OneCycle
Concept: Plateau watches test scores (validation loss/accuracy), not the clock. After patience intervals with no improvement, multiply by factor (e.g. 0.1). OneCycle briefly raises then lowers in one short cycle.
Analogy: Plateau = GPS says “slow down” only in a jam; OneCycle = a short sprint course — accelerate to explore, then glide to a stop.
Tip: Plateau is safe when data/model size is uncertain. OneCycle suits short practice runs. With Ch.02 optimizer you get direction and speed.
Schedulers at a glance
Step / MultiStep — every N intervals (or checkpoint): . is often 0.1. Common for long training.
Cosine Annealing — . Smooth late fine-tuning.
Warmup — linear rise early: . Eases early error spikes on large models.
ReduceLROnPlateau — if no val improvement for patience epochs: . Brake only when stuck.
Why it matters
1. Better results in the same training time
Good schedules often yield lower error and higher accuracy for the same training time. Without late braking, final polish is hard; shrinking stride late helps land on the minimum. Faster convergence saves time and experiment rounds.
2. Completes the Ch.02 optimizer
The optimizer sets direction and per-spot stride, but shrinking overall stride over time is the scheduler’s job. No warmup + huge early → error explosions; no late decay → endless zigzag around the minimum. Many “stuck” runs converge after adding one schedule.
3. First graph to check when scores stall
The stride (LR) plot tells a story. Perfectly flat (━) → schedule not updating or wrong timing; drops too early → under-learning; stays high → late oscillation. Check scheduler with Ch.02 lr and optimizer first.
How it is used
① Start, middle, end — stride must change
Like exploring a town: main street → alleys → one step at the door. Training has explore → converge → land acts. Large early to scan widely, then brake, then small to polish near the minimum. The scheduler is the timer for which act you are in.
② Each scheduler has a personality — pick for the job
Step = elevator cuts at set times (long training). Cosine = smooth late decay (long runs, text AI). Warmup = engine warm-up only early (large models). Plateau = acts only on score stall. OneCycle = short practice runs. Pick the driving style for your situation.
③ The golden stride rhythm — order matters
Like hiking: ① check how steep it is (loss/error) → ② take one step that way (Ch.02 optimizer) → ③ set the next stride (scheduler). Plateau-style schedules watch validation scores and brake only when stuck.
④ Read stride and error plots together
When error stalls and drops right after you shrink stride, scheduling worked. Too early braking → under-learning.
Symptom-first: early error spike → lower stride + Warmup; late zigzag → Cosine/Step/Plateau; flat stride plot (━) → check schedule updates. Change one thing at a time and take notes.
Summary
One line: Scheduling adjusts stride () over training for fast exploration early and precise landing late.
Key schedulers: Step, Cosine, Warmup, Plateau, OneCycle — pick for your scenario.
Checks (symptom → look first)
- Early error spike: lower stride, Warmup
- Late zigzag / no convergence: Cosine/Step/Plateau, , patience
- Flat stride plot (━): schedule not updating, interval confusion
- Scores stall only: Plateau, data/model settings
Tuning order: 1) notes on error + stride → 2) base stride + Warmup → 3) Ch.02 optimizer → 4) schedule → 5) change one thing at a time
Ch.02 optimizer (compass) + Ch.03 scheduler (brake & gas) = the everyday stack.
How to approach problems
Learning rate scheduling changes the stride (learning rate) the Ch.02 optimizer uses over time or scores. Step cuts at fixed intervals; Cosine decays smoothly; Warmup raises stride early to avoid shock; Plateau acts only when scores stall. One stride cycle: check error → move (optimizer) → set next stride (scheduler); Plateau also watches validation. Early error spike → Warmup; late zigzag → Cosine/Step/Plateau; flat stride plot (━) → schedule not updating.
Example (definition)
"In step decay, what is the factor multiplied to every N intervals?
① patience
② gamma
③ beta"
The decay factor is (gamma). → Answer 2
Example (scenario)
"Text AI training shows unstable error early. With Ch.02 optimizer, which combo to check first?
① warm-up + smooth decay
② infinite stride increase
③ remove scheduler only"
Early instability → try warm-up + smooth decay first. → Answer 1
Example (calculation)
With , , after 2 Step decays, what is (×1000, integer)?
→ ×1000 = 1. → Answer 1
Definition example — "What is the main purpose of Warmup?
① data augmentation
② ease early instability from large LR
③ zero weights" → Warmup slowly raises early to reduce shock. Answer 2
True/False example — "ReduceLROnPlateau uses validation metrics." → True. Answer 1
Application example — "Validation loss fails to improve for 5 consecutive epochs" → Plateau acts on validation stall. Answer 1
Choice example — "Which gives smoother late decay?
① StepLR only
② CosineAnnealing" → Cosine decays smoothly along a cos curve. Answer 2
Concept example — "Why is Ch.02 optimizer + scheduler common?
① direction and time-varying stride complement
② replaces backprop" → Optimizer and scheduler complement each other. Answer 1
Calculation example — "128 samples, batch size 32 — steps per epoch (integer)?" → . Answer 4