Ch.06
Batch & Layer Normalization
When you study deep learning, you'll often run into the word "normalization." In many languages the terms sound similar, but regularization against overfitting (Regularization — e.g. L2 penalty, dropout) and normalization that matches signal size per layer (Normalization — e.g. batch norm, layer norm) play completely different roles. This chapter goes deep on normalization layers that keep each layer's activations from growing too large or too small so the model can train steadily without wobbling.
Picture an orchestra. If the violin is too loud and the piano too quiet, even a great conductor can't produce beautiful music. Deep neural networks are the same: when signal size swings layer by layer, learning slows or stops entirely. Batch Normalization groups several samples into one reference; Layer Normalization sets reference inside a single sample. We explain how train vs serve behavior differs, step by step, with beginner-friendly analogies.
The diagram shows batch norm, layer norm, train/inference, and role distinction as before → after in plain language.
Left = before, right = after. Batch norm groups samples; layer norm works inside one sample. Training uses the current batch; serving uses stored reference. Overfitting tools (L2/dropout) handle memorization/weights; norm layers handle signal size.
Match signal size per layer
Stabilize training; tell roles apart
Normalization Layers: Matching Scale Per Layer
1. Why match scale per layer?
Plain language: A neural network passes data through many layers to find the answer. When early-layer weights shift slightly during training, the size and spread of data reaching later layers can snowball into big changes. When signals wobble layer by layer (internal covariate shift), the model gets confused and can't learn properly.
Analogy: In a relay race, the next runner needs a steady baton handoff. If the first runner sprints one lap and jogs the next, timing breaks. Norm layers keep each layer's signal "stride" steady so the next layer always receives a stable range.
Tip: If loss suddenly spikes or gradients vanish, before only tweaking learning rate, check norm layer placement and train/inference mode.
2. Batch Normalization — align several samples at once
Plain language: Data usually enters the model in mini-batches. Batch norm uses the whole batch as reference: how far is each value from the batch mean? Fixing everything rigidly to 0–1 can hurt expressiveness, so learnable parameters (, ) are multiplied and added to flexibly rescale and shift again.
Core formula:
One-line meaning: Subtract batch mean from input and divide by standard deviation . is a tiny safety value so the denominator never hits zero.
Analogy: After a test, you don't just look at raw score — you use class mean and standard deviation to see where you stand (like a z-score).
Tip: Very effective in vision CNNs when you can feed 32 or 64 images per batch; usually placed right after Conv or Linear layers.
3. Layer Normalization — balance inside one sample
Plain language: Batch norm needs other samples as reference — a downside. Layer norm ignores other data and uses only the feature values inside one sample to compute mean and variance. It shines when batch size is 1 or sentence length varies (Transformer, RNN).
Analogy: Batch norm is like the class math average; layer norm is one student's Korean, English, math, science scores averaged within that student — ignoring the desk neighbor.
Tip: Standard in Transformer blocks (like ChatGPT's backbone) and RNN/LSTM. When memory forces mini-batch size 1–2, layer norm is safer than batch norm.
4. Training vs inference — batch norm's two faces
Plain language: Batch norm behaves differently while training vs at serve time. While training, it normalizes using the current mini-batch while quietly recording running mean and spread across all data seen so far. At serve time, even if the user uploads one image, it uses statistics accumulated during training.
Analogy: National-team practice sets intensity from today's squad; the World Cup match picks players and tactics from experience built over many friendlies.
Tip: In PyTorch, `model.train()` and `model.eval()` switch modes. Forgetting `model.eval()` at deploy means each upload shifts the reference and yields wrong results. Always enable inference mode before shipping.
Core concepts and formulas at a glance
Batch Normalization — balance using the mini-batch as reference.
Plain language: If a middle layer suddenly outputs abnormally large or small values, the next layer gets shocked. Batch norm gathers values from one input mini-batch and adjusts balance — "you're above our batch average, turn down; you're below, turn up."
Key point: Strongest when you can feed 32+ samples at once. With only 1–2 per step, the reference shakes and it can hurt.
Everyday analogy: After a test, scoring yourself vs whole-class average — like a standard score.
Layer Normalization — balance independently inside one sample.
Plain language: Variable-length chat data is hard to batch neatly. Layer norm ignores other samples and averages inside one sample (one sentence) only.
Key point: Works perfectly with batch size 1. A core part of Transformer-based LLMs.
Everyday analogy: Instead of comparing to others, average your own subject scores and balance within yourself.
Training vs inference — practice and live play must differ.
Plain language: Batch norm uses current mini-batch stats while training but also secretly tracks cumulative mean across all data. At serve time, even one image uses that accumulated reference.
Key point: Always call PyTorch's `model.eval()` before deploy. Otherwise predictions keep changing — a major bug.
Everyday analogy: Practice matches today's squad; World Cup matches rely on experience from dozens of games.
Two "normalizations" — how to tell them apart
Plain language: Same Korean label, different jobs. Regularization (L2, dropout) stops rigid memorization. Norm layers (batch/layer norm) keep internal data pressure steady so training doesn't blow up.
Key point: High train, low val → regularization. NaN or wild loss during training → norm layers.
Everyday analogy: Regularization = lightening your backpack (weight control); norm layers = mixing orchestra volume (signal size).
Why does this matter?
1. Makes imagined deep networks actually trainable
In theory, stacking hundreds of layers should make models smarter, but activations and gradients used to explode or vanish so training was impossible. Batch and layer norm tame that instability — the top enabler for bolder learning rates and much deeper models in practice.
2. Separates two "normalizations" that sound alike
In Korean both get called "normalization," which confuses beginners. L2 and dropout (Regularization) stop overfitting and slim down weights. Batch/layer norm (Normalization) polishes signal size flowing between layers. Practice scores great but real test fails → former; model sputters from the start with weird loss → latter.
3. Pick the right fit for data and model type
Even great techniques need the right context. Batch norm fits image CNNs; forcing it into Transformer or RNN text models can break training. Understand batch size, sequence length, etc., and choose the right norm layer for best performance.
How it's used in practice
① Diagnose: is our model's engine sputtering?
Everyday example: If your car engine shakes and the speedometer swings wildly, you can't drive normally. In deep learning, if loss jumps around or you see `NaN` (not a number), the data pipeline is broken. Don't just floor the accelerator (learning rate) — add norm layers in the right place like an oil change.
If train accuracy is 99% but new data only hits 50%, that's overfitting. Add dropout or data augmentation (Regularization), not only norm layers (Normalization).
② Photos & vision: batch norm's main stage
Everyday example: Building a phone AI to tell dogs from cats — with enough GPU memory you might feed 32 or 64 photos per step. Batch norm works great by averaging brightness and feature spread across "this pile of 64 photos."
High-res medical images where you only fit 1–2 photos per step? The reference keeps shaking and batch norm backfires. Try larger effective batch size or another norm approach.
③ Chatbots & translators: layer norm's solo stage
Everyday example: User messages range from "Hi?" (two characters) to "Summarize this article in three lines" (very long). Forcing variable-length sentences into one batch average is awkward.
Text models use layer norm to adjust signal size only inside the one sentence being analyzed, ignoring neighbors — like a marathoner keeping their own pace.
④ Pre-launch checklist: turn on inference mode (eval)
Everyday example: You trained hard and launched a web service, but results swing every time a user uploads an image. Cause: the model never switched to "performance mode (inference)."
One `model.eval()` in PyTorch stops computing batch means from new data and uses reliable stats accumulated during training. The crucial last step: drop practice habits and follow the live playbook.
Three-line takeaway
This chapter's core is batch and layer normalization — keeping signal size steady layer by layer to stabilize training. Don't confuse them with regularization against overfitting (L2, dropout) despite similar naming. Regularization stops blind memorization; normalization layers keep water pressure (signal size) even in the data pipeline.
Batch norm groups several samples (mini-batch) to set reference — common in image CNNs; at serve time you must use stats accumulated during training. Layer norm sets reference inside one sample only — essential for NLP (Transformer) and very small batches.
Practice tip: if training refuses to run and loss explodes, calm it with norm layers (Normalization); if validation keeps dropping while train stays high, use regularization techniques — prescribe by situation and role.
Problem-solving guide
Problems cluster around unstable training, batch vs layer norm, and two kinds of "normalization".
When training wobbles, check NaN loss, exploding/vanishing gradients, and wild train loss swings. Before going deeper, check norm layer placement and train/inference mode.
Train good, val bad suggests overfitting (memorization). Norm layers alone may not fix it — also consider L2, dropout, early stopping, augmentation.
Batch norm groups samples (batch 32+). Layer norm works inside one sample (batch=1, Transformers). Training = current batch + stored experience; inference = stored reference — call `model.eval()` before deploy.
Clues: batch=1/Transformer → layer norm; inference mode/stored stats → batch norm; NaN → norm layers; weight penalty/neuron off → overfitting tools (not this chapter's main answer).
Definition problems — match option numbers to roles first.
"Core goal of norm layers?" →
① weight L2 penalty ·
③ neuron off = overfitting prevention.
② stabilize signal size per layer is this chapter's core → Answer 2.
"Batch norm reference?" →
① group of samples (mini-batch) is batch norm.
② inside one sample is layer norm → for batch norm questions Answer 1.
"Batch norm at inference?" →
① uses stored mean/spread from training.
② current batch only fits training → Answer 1.
Scenario problems — read the situation first.
NaN or unstable gradients →
② add norm layer before
① going deeper → Answer 2.
Transformer · batch=1 →
② layer norm fits naturally → Answer 2.
Shaky after deploy →
① check inference mode (`model.eval()`) first → Answer 1.
Deploying while `model.train()` stays on lets batch norm use different mini-batch stats each time, so results wobble → Answer 2.
Calculation problems — plug in step by step.
Average 2, value 5, spread 9 → √9=3, (5−2)÷3 = 1 → option
② → Answer 2.
Normalized −1, ×4 +2 → −4+2 = −2 → Answer 2.
True/False — "Inference always uses only the current mini-batch" is a common mistake. Inference uses stored reference → false (0).
"Batch norm and L2 share the same main goal" → L2 = weights/memorization; batch norm = signal size → false (0).
"Batch norm can update running mean and running variance during training" → correct → true
(1) .
"Tiny batch shakes batch norm reference" → correct → true
(1) .
Multiple choice — "Align by distance from average" → core norm-layer form → Answer 1.
"Overfitting prevention vs norm layers" → not
① same or
③ unrelated; different goals
② (weights/memorization vs signal size) → Answer 2.
Concept problems — batch=1 with batch norm only → unstable; layer norm
② → Answer 2.
Text/Transformer models → layer norm
② is standard → Answer 2.
Calculation problems — normalized 2, ×3 +1 → 6+1 = 7 → Answer 3.
Norm using stored reference at inference → batch norm
② → Answer 2.