Everyone's AI
Machine learningPlayground
Loading...

Learn

Ch.09

Jacobian Matrix: First Derivatives of Multivariable Vector Functions

Math diagram by chapter

Select a chapter to see its diagram below. View the flow of intermediate math at a glance.

Jacobian: when multiple inputs move, what about the output?

Ch.08 gradient for one output; Ch.09 Jacobian JJJ for several. Animation: left input → right output. First warp (fff), then linear patch (JJJ).

1. Input grid2. f: warp3. Small ▱4. J: linear
InputOutputfJ→ Far: grid warps (nonlinear f)

Red grid = coordinates, green arrow = map, orange ▱ = small patch under fff, purple ▱ = JJJ approximation. Similar ▱ shapes mean f≈JΔxf\approx J\Delta\mathbf{x}f≈JΔx.

Picture a huge factory machine with many dials (inputs) and many gauges on the dashboard (outputs). Nudge dials 1 and 2 together—how much do gauges 3 and 4 move?
Ch.08 gradient was a single compass when there is only one output (e.g. loss)—the steepest uphill direction. The Jacobian matrix (JJJ) is the ultimate sensitivity dashboard: one table capturing every tiny interaction between all inputs and all outputs.
Our world and deep-learning models look like wrinkled, unpredictable nonlinear surfaces f\mathbf{f}f from far away. Zoom in at one point, though, and the surface looks like straight lines and flat planes. The Jacobian mimics that instant as a parallelogram grid (local linear approximation)—the mathematical tool that lets huge neural networks trace error and tune themselves.

Jacobian Matrix: When Multiple Inputs Move, What Happens to the Output?

① Gradient vs Jacobian — one result vs many
Ch.08 gradient is when there is one answer: “If I study 10 more minutes, how much does my score go up?” The Jacobian is when there are several answers: “If I nudge the shoulder and elbow a little, how much do the hand’s X and Y each move?”—all in one table. Ch.08 = one compass; Ch.09 = a big table linking many dials to many gauges.
② Reading one cell JijJ_{ij}Jij​ — “dial j → needle i”
JijJ_{ij}Jij​ looks scary but means simply: “If I turn input jjj a little, how much does output iii move?” Rows = outputs, columns = inputs. In a 2×2 example, row 1 lists how the first output reacts to each input.
③ Δy≈JΔx\Delta\mathbf{y}\approx J\Delta\mathbf{x}Δy≈JΔx — “nudge in → about this much out”
If inputs move a tiny bit (Δx\Delta\mathbf{x}Δx), outputs move a tiny bit too (Δy\Delta\mathbf{y}Δy). Roughly: output change ≈ Jacobian × input change. A winding road looks curved from far away, but near one point it looks like a short straight line—that is what the top vs bottom panels show.
④ Backprop — multiply tables to send error backward
A network has many layers. When the last layer says “wrong,” that signal must travel backward so we know what to fix. Each layer has a small Jacobian table; we multiply them (Jtotal=JL⋯J1J_{\text{total}}=J_L\cdots J_1Jtotal​=JL​⋯J1​). Think table × table × table, like linking LEGO blocks in reverse.
⑤ det⁡J\det JdetJ — how much area grew · robot-arm warning
A small square patch becomes a parallelogram; det⁡J\det JdetJ tells how many times the area scaled (Ch.05 determinant). ∣det⁡J∣=6|\det J|=6∣detJ∣=6 → 6× area. If a robot arm is fully straight, det⁡J=0\det J=0detJ=0 — a paralysis (singularity) warning: it cannot move even 1 mm in some directions.
1. Vector-valued functions: many input sticks, many output screens
Until now we often had scalar functions—turn many dials but only one number (loss) comes out. But pixel coordinates, hundreds of neuron values in a hidden layer, and similar outputs arrive as vectors far more often. When nnn inputs feed mmm outputs at once, we call that machine a vector-valued function f:Rn→Rm\mathbf{f}:\mathbb{R}^n\to\mathbb{R}^mf:Rn→Rm. Nudge one input stick and dozens of output screens wiggle together—each by a different amount.
2. Local linearization: Earth looks flat under a magnifier
Like the top panel of the visual, real change warps grids into complex curves (nonlinear). What if we zoom in thousands of times near one point? Curves look like short straight lines; curved surfaces like flat planes. Calculus lives in that narrow local patch; the Jacobian is the slope table for that flattened map.
3. Definition: the ultimate change summary
Jij=∂fi∂xjJ_{ij}=\frac{\partial f_i}{\partial x_j}Jij​=∂xj​∂fi​​ looks heavy, but means simply: “If I turn dial jjj one notch, how many notches does gauge iii move?” Write every such number in row iii, column jjj. Then many small dial turns Δx\Delta\mathbf{x}Δx produce output change Δy\Delta\mathbf{y}Δy via one clean line: Δy≈J Δx\Delta\mathbf{y}\approx J\,\Delta\mathbf{x}Δy≈JΔx.
4. Link to Ch.08 gradient: compasses stacked into a tower
With one output, JJJ is a single row—the gradient as a row vector. With three outputs, stack three gradient rows. The Jacobian is every output’s gradient, row by row, in one big compass collection.
5. Geometry: squashed area scale (det⁡J\det JdetJ)
A small square input patch becomes a parallelogram under JJJ. det⁡J\det JdetJ (Ch.05) tells how many times the area grew or shrank—like clay stretching. ∣det⁡J∣=6|\det J|=6∣detJ∣=6 means unit square area becomes 6×.
In one line: The Jacobian JJJ is the ultimate sensitivity dashboard—how all dials move all gauges. Locally, Δy≈JΔx\Delta\mathbf{y}\approx J\Delta\mathbf{x}Δy≈JΔx straightens complex change; in deep learning it is the mathematical heart of backprop through stacked layers.
Deep learning is a factory woven from Jacobians. After ReLU, sigmoid, and other nonlinearities, data bends—but backprop uses local linearization to compute each layer’s Jacobian. Ch.08 gradient was a compass for the final loss only; between layers we need matrices capturing how thousands of neurons influence one another.
Learning is nothing but the chain rule in action: keep multiplying Jacobian matrices one after another. GAN generators that create human-like faces and autoencoders that compress data both ask, “If I twist the latent noise I fed in just a little, how does the image in front of me warp and change?”—and read the answer through the Jacobian JJJ to decide which way to learn.
1. Backpropagation: error flowing upstream
To shrink loss, propagate error from the last layer back to the inputs. With dozens of layers, multiply local Jacobians like stepping stones (Jtotal=JL⋯J1J_{\text{total}}=J_L\cdots J_1Jtotal​=JL​⋯J1​)—even in huge models you get full sensitivity and can update weights.
2. Normalizing flows: density stretch and squeeze
Generative AI can “knead” simple clay (a Gaussian) into a detailed sculpture (a complex data distribution). Stretch space 2× and density halves. log⁡∣det⁡J∣\log|\det J|log∣detJ∣ corrects probability when volume changes—the same picture as Ch.05 determinants.
3. Robot arms and singular (frozen) poses
Imagine a shoulder-and-elbow arm. The Jacobian answers: “If I rotate a joint 1°, how much does the hand move in X and Y?” If the arm is fully straight, det⁡J=0\det J=0detJ=0: the arm cannot move even 1 mm in some directions—a singularity (paralysis) warning in real robotics.
When solving, follow these steps
① Table size — how many outputs? inputs?
With mmm outputs and nnn inputs, JJJ is an m×nm \times nm×n table. Rows = outputs, columns = inputs (gauge number / dial number).
② Fill each cell — "dial j → needle i"
Cell (i,j)(i,j)(i,j) holds "if input jjj moves a tiny bit, how much does output iii change?" The formal name is partial derivative (JijJ_{ij}Jij​), but think one reaction size.
③ Small moves — tiny input → tiny output
When inputs change just a little (Δx\Delta\mathbf{x}Δx), output change (Δy\Delta\mathbf{y}Δy) is roughly JJJ × input change: Δy≈JΔx\Delta\mathbf{y}\approx J\Delta\mathbf{x}Δy≈JΔx.
④ One output? Ch.08 gradient
If there is only one result (m=1m=1m=1), JJJ is Ch.08 gradient ∇f\nabla f∇f laid on its side (∇fT\nabla f^\mathsf{T}∇fT).
3-step workflow:
① check size →
② fill partials →
③ if functions compose, multiply JgJfJ_g J_fJg​Jf​
  • In wordsJacobian
  • MeaningJij=∂fi/∂xjJ_{ij}=\partial f_i/\partial x_jJij​=∂fi​/∂xj​, size m×nm\times nm×n
  • In wordsLocal approx
  • MeaningΔy≈JΔx\Delta\mathbf{y}\approx J\Delta\mathbf{x}Δy≈JΔx
  • In wordsGradient
  • Meaningm=1m=1m=1: J=∇fTJ=\nabla f^\mathsf{T}J=∇fT
  • In wordsChain
  • MeaningJg∘f=JgJfJ_{g\circ f}=J_g J_fJg∘f​=Jg​Jf​
  • In wordsdet J
  • Meaningarea/volume scale (Ch.05)
  • In wordsBackprop
  • Meaning∂L/∂x=JT(∂L/∂f)\partial L/\partial\mathbf{x}=J^\mathsf{T}(\partial L/\partial\mathbf{f})∂L/∂x=JT(∂L/∂f)
In wordsMeaning
JacobianJij=∂fi/∂xjJ_{ij}=\partial f_i/\partial x_jJij​=∂fi​/∂xj​, size m×nm\times nm×n
Local approxΔy≈JΔx\Delta\mathbf{y}\approx J\Delta\mathbf{x}Δy≈JΔx
Gradientm=1m=1m=1: J=∇fTJ=\nabla f^\mathsf{T}J=∇fT
ChainJg∘f=JgJfJ_{g\circ f}=J_g J_fJg∘f​=Jg​Jf​
det Jarea/volume scale (Ch.05)
Backprop∂L/∂x=JT(∂L/∂f)\partial L/\partial\mathbf{x}=J^\mathsf{T}(\partial L/\partial\mathbf{f})∂L/∂x=JT(∂L/∂f)
① rows=outputs.
② linear f=Axf=Axf=Ax ⇒ J=AJ=AJ=A.
③ small Δx\Delta\mathbf{x}Δx.
④ singular ⇒ collapse one way.

Worked examples

Ex 1 — 2×2 J
Problem: f(x,y)=(x+y,  x−y)\mathbf{f}(x,y)=(x+y,\;x-y)f(x,y)=(x+y,x−y). Find JJJ.
Solution: f1=x+y⇒(1,1)f_1=x+y\Rightarrow(1,1)f1​=x+y⇒(1,1), f2=x−y⇒(1,−1)f_2=x-y\Rightarrow(1,-1)f2​=x−y⇒(1,−1). J=(111−1)J=\begin{pmatrix}1&1\\1&-1\end{pmatrix}J=(11​1−1​).
Ex 2 — at a point
Problem: f(x,y)=(x2,  y)\mathbf{f}(x,y)=(x^2,\;y)f(x,y)=(x2,y) at (1,0)(1,0)(1,0). Find JJJ.
Solution: ∂f1/∂x=2x=2\partial f_1/\partial x=2x=2∂f1​/∂x=2x=2, others 0; ∂f2/∂y=1\partial f_2/\partial y=1∂f2​/∂y=1. J=(2001)J=\begin{pmatrix}2&0\\0&1\end{pmatrix}J=(20​01​).
Ex 3 — linear
Problem: f(x)=Ax\mathbf{f}(\mathbf{x})=A\mathbf{x}f(x)=Ax. Find JJJ.
Solution: AAA (same everywhere).
Ex 4 — chain
Problem: f(x)=xf(x)=xf(x)=x, g(u)=2ug(u)=2ug(u)=2u. Find Jg∘fJ_{g\circ f}Jg∘f​ (1×1).
Solution: Jf=1J_f=1Jf​=1, Jg=2J_g=2Jg​=2, Jg∘f=JgJf=2J_{g\circ f}=J_g J_f=2Jg∘f​=Jg​Jf​=2.
Ex 5 — det J
Problem: J=(2003)J=\begin{pmatrix}2&0\\0&3\end{pmatrix}J=(20​03​). Unit square area?
Solution: ∣det⁡J∣=∣6∣=6|\det J|=|6|=6∣detJ∣=∣6∣=6 → 6×.
Ex 6 — linear layer
Problem: z=Wx+b\mathbf{z}=W\mathbf{x}+\mathbf{b}z=Wx+b. Find JJJ.
Solution: linear map, so WWW.

Practice

When m=1m=1m=1, Jacobian JJJ matches putting gradient ∇f\nabla f∇f in a row.
1 / 5