Ch.01

Weight Initialization

Weight initialization is choosing the initial values of each layer's weights and biases before training. A bad start leads to vanishing or exploding gradients and often makes learning nearly impossible; a good start leads to faster convergence and stable training. This chapter covers the concept of initialization, the intuition and formulas behind Xavier and He initialization, and how they are used in practice.

Weight Initialization: A Good Start Is Half the Battle

What is weight initialization? — Each layer of a neural network has weights $W$ and biases $b$ . Before training, these values are undefined, so we must choose what numbers to use initially. This process is called weight initialization. Intuitively, it is like choosing where to start a marathon: too far back (weights too small) and progress is slow; too far forward (weights too large) and training can explode and diverge.Mathematically — In one layer the linear combination is

z = W \mathbf{x} + b

, where

\mathbf{x}

is the input vector,

W

the weight matrix, and

b

the bias. If all elements of

W

are zero, every neuron in that layer gives the same output, symmetry is preserved, and gradients do not spread properly during backprop. So we usually initialize with small random numbers, but the distribution (scale) of those numbers matters. We adjust variance using the layer's input dimension

n_{in}

and output dimension

n_{out}

so that activations do not grow or shrink too much as they pass through.In practice — With bad initialization, a spam classifier may show almost no decrease in loss or NaNs. In deep CNNs (e.g. medical imaging or fraud detection), skipping Xavier or He often leads to vanishing gradients in early layers and training appears stuck. If the scale is too large, gradients explode and training becomes unstable. So in practice Xavier (for tanh·sigmoid) or He (for ReLU) initialization is standard.

W

Effect of initialization scale

Good initialization sets W, b scale so that variance is preserved across layers.

A good start is half the battleProper initialization → fast convergence · stable training

W

Why it matters

Vanishing and exploding gradients — In deeper networks, backpropagated gradients are products of many numbers (chain rule). If weights are too small, this product tends to zero (vanishing gradient) and early layers barely update; if too large, it explodes (exploding gradient) and you get NaN or Inf. Good initialization keeps variance stable across layers so that gradients stay at a reasonable scale even in deep networks.Convergence speed — With proper initialization you start at a good point on the loss surface. A bad starting point can trap you in poor local minima or make convergence very slow. In practice, initialization is tuned together with learning rate by monitoring validation loss.

How it is used

Xavier (Glorot) initialization — So that the variance of

z

does not depend on input/output size,

W

is sampled from a uniform distribution

U(-\sqrt{6/(n_{in}+n_{out})},\ \sqrt{6/(n_{in}+n_{out})})

or a normal distribution

\mathcal{N}(0,\ \sigma^2)

with

\sigma^2 = 2/(n_{in}+n_{out})

. It fits symmetric activations like tanh and sigmoid.He initialization — ReLU zeros out negative inputs, so output variance is about half the input variance. He initialization uses

\sigma^2 = 2/n_{in}

to compensate. It is the default in modern CNNs and MLPs that use ReLU or Leaky ReLU.Practical choice — Use He for ReLU-family activations and Xavier for tanh·sigmoid. Frameworks (PyTorch, TensorFlow) usually apply one of these by default depending on the layer type.

Summary

Weight initialization is the process of setting the initial values of each layer's weights and biases before training. Initializing everything to zero keeps neurons symmetric and prevents proper learning; random values that are too large or too small lead to exploding or vanishing activations and gradients. Xavier and He initialization adjust variance based on layer size and are widely used: Xavier for symmetric activations like tanh·sigmoid, He for ReLU-family. A good start reduces vanishing and exploding gradients and makes convergence faster and more stable.

Problem-solving guide

Summary — Weight initialization is the step of choosing initial values for each layer's

W

and

b

before training. Zero initialization breaks learning due to symmetry, so we usually use small random numbers and adjust variance (scale). Xavier uses

\sigma^2 = 2/(n_{in}+n_{out})

for tanh·sigmoid; He uses

\sigma^2 = 2/n_{in}

for ReLU-family. Good initialization reduces vanishing/exploding gradients and speeds convergence.

Type $Weight init definition$
Solution / example (keyword → answer) $W$

Type $Xavier$
Solution / example (keyword → answer) $\sigma^2 = 2/(n_{in}+n_{out})$

Type $He$
Solution / example (keyword → answer) $\sigma^2 = 2/n_{in}$

Type $Vanishing gradient$
Solution / example (keyword → answer) $Gradient nears 0 when weights too small. \to 1$

Type $Exploding gradient$
Solution / example (keyword → answer) $Gradient explodes when weights too large. \to 2$

Type $Xavier uniform$
Solution / example (keyword → answer) $[-\sqrt{6/(n_{in}+n_{out})},\ \sqrt{6/(n_{in}+n_{out})}]$

Type $Layer size$
Solution / example (keyword → answer) $n_{in}=4$

Type	Solution / example (keyword → answer)
Weight init definition	Setting $W$ , $b$ before training. Zero init not recommended. → concept choice 1
Xavier	$\sigma^2 = 2/(n_{in}+n_{out})$ , tanh·sigmoid. → 1
He	$\sigma^2 = 2/n_{in}$ , ReLU family. → 2
Vanishing gradient	Gradient nears 0 when weights too small. → 1
Exploding gradient	Gradient explodes when weights too large. → 2
Xavier uniform	Range $[-\sqrt{6/(n_{in}+n_{out})},\ \sqrt{6/(n_{in}+n_{out})}]$ . Use integer when computing.
Layer size	$n_{in}=4$ , $n_{out}=6$ → $n_{in}+n_{out}=10$ .

Example (definition)

"What is the main purpose of weight initialization?

① Match layer scale before training

② Increase learning rate

③ Data augmentation"

Purpose is to keep activation and gradient scale stable across layers. → Answer 1

Example (Xavier vs He)

"Common initialization for layers using ReLU?

① Xavier

② He

③ Zero"

He is used for ReLU-family. → Answer 2

Example (calculation)

When

n_{in}=4

n_{out}=6

, what is

n_{in}+n_{out}

(integer) in Xavier?

4+6=10

. → Answer 10

Definition example — "What is the main purpose of weight initialization?

① Match layer scale before training

② Increase learning rate

③ Data augmentation" → Purpose is to keep scale stable across layers. Answer 1

True/False example — "Weight initialization is the process of setting

W

b

before training." → True. Answer 1

Scenario example — "When loss barely decreases in a spam classifier, what to check first?

① Initialization·learning rate

② Data size only

③ Batch size only" → Check initialization·learning rate first. Answer 1

Choice example — "In He initialization,

\sigma^2

is?

①

2/n_{in}

②

2/(n_{in}+n_{out})

③

1/n_{in}

" → He uses

\sigma^2=2/n_{in}

. Answer 1

Concept example — "In Xavier, if

n_{in}+n_{out}=6

, the value (integer) of

6/(n_{in}+n_{out})

is?

① 1

② 2

③ 3" →

6/6=1

. Answer 1

Calc example — "When

n_{in}=4

n_{out}=6

, the value (integer) of

n_{in}+n_{out}

in Xavier is?" →

4+6=10

. Answer 10

Weight Initialization: A Good Start Is Half the Battle

z = W \mathbf{x} + b

, where

\mathbf{x}

is the input vector,

W

the weight matrix, and

b

the bias. If all elements of

W

n_{in}

and output dimension

n_{out}

Type

Solution / example (keyword → answer)

Weight init definition

Setting

W

b

before training. Zero init not recommended. → concept choice 1

Xavier

\sigma^2 = 2/(n_{in}+n_{out})

, tanh·sigmoid. → 1

\sigma^2 = 2/n_{in}

, ReLU family. → 2

Vanishing gradient

Gradient nears 0 when weights too small. → 1

Exploding gradient

Gradient explodes when weights too large. → 2

Xavier uniform

Range

[-\sqrt{6/(n_{in}+n_{out})},\ \sqrt{6/(n_{in}+n_{out})}]

. Use integer when computing.

Layer size

n_{in}=4

n_{out}=6

→

n_{in}+n_{out}=10