Everyone's AI
Machine learningPlayground
Loading...

Learn

Ch.11

Taylor Series: Approximating Complex Functions with Polynomials

Math diagram by chapter

Select a chapter to see its diagram below. View the flow of intermediate math at a glance.

Even a complex curve looks like a line (1st order) or parabola (2nd order) when you zoom in near one point. Watch below how raising the degree makes the approximation stick to the original function.
Taylor series stores how the function moves at this point using derivatives, and draws a polynomial that matches nearby values.
1st order: follow with tangent2nd order: bend like a parabola3rd order: match farther out
지금 보는 단계

1st order: follow with tangent

원함수1차 근사핵심: 중심에서 멀수록 오차 큼

핵심: 차수가 올라갈수록 보통 더 정확해집니다.

Purple is the original function; orange, green, and blue are 1st, 2nd, and 3rd-order Taylor approximations. Error grows as you move away from the center.

A Taylor series rewrites a complicated function as a polynomial near the point where you are standing. A first-order approximation follows like a tangent line; second and third orders hug the curve more tightly. In AI, this idea appears when we approximate loss or activation functions and when we understand Newton’s method and second-order optimization.

Taylor Series: Following Complex Functions with Polynomials

What is a Taylor series? It is a formula that replaces a difficult function f(x)f(x)f(x) with a polynomial valid near a single point aaa. It combines the value, slope, and curvature at aaa so that nearby xxx values are almost the same as the original function.
When you zoom a map, a curved road looks straight. Taylor series works the same way: zoom near aaa and a complex function behaves like 1+x+x2/2+⋯1 + x + x^2/2 + \cdots1+x+x2/2+⋯. Use only 1st order and you get the tangent; go to 2nd order and you get something close to a parabola.
Formally, Tn(x)=f(a)+f′(a)(x−a)+f′′(a)2!(x−a)2+⋯+f(n)(a)n!(x−a)nT_n(x)=f(a)+f^{\prime}(a)(x-a)+\frac{f^{\prime\prime}(a)}{2!}(x-a)^2+\cdots+\frac{f^{(n)}(a)}{n!}(x-a)^nTn​(x)=f(a)+f′(a)(x−a)+2!f′′(a)​(x−a)2+⋯+n!f(n)(a)​(x−a)n. Here f(k)(a)f^{(k)}(a)f(k)(a) is the kkkth derivative at aaa and k!k!k! is kkk factorial. A Maclaurin series is the special case a=0a=0a=0. Higher nnn usually gives a better fit farther from aaa.
In machine learning, approximating the loss quadratically near parameters links to Newton’s method and the Hessian (Ch.10). Activation functions can also be read as linear or quadratic on small intervals, helping you reason about learning rate and approximation error.
In short, Taylor series approximates complicated functions with polynomials near a point. 1st order connects to gradients; 2nd order connects to the Hessian and Newton; higher order improves accuracy. Together with Ch.08 gradients and Ch.10 Hessian, it forms the math backbone of optimization.
Derivatives alone tell the slope now, not the whole curve. Taylor series uses higher derivatives at the same point to summarize local shape in one polynomial—ideal for optimization, error analysis, and numerical computation.
Ch.10’s Hessian generalizes the quadratic term f′′(a)2(x−a)2\frac{f^{\prime\prime}(a)}{2}(x-a)^22f′′(a)​(x−a)2 to many variables. Understanding 2nd-order Taylor explains why second derivatives drive curvature and Newton steps.
1. How is it used in gradient descent? (1st-order approximation)
Gradient descent moves little by little in the direction that reduces loss the fastest at the current point. This is exactly the first-order Taylor idea: near the current point, we treat the loss like a tangent line and choose the next step from that local linear view.
A practical flow is:
- compute the current loss and gradient,
- use the 1st-order approximation to decide a descending direction,
- move one step with the learning rate.
This view matters because the update rule is not random memorization; it is a decision grounded in first-order approximation. It also explains why too large a learning rate can oscillate and why too small a one can be slow.
2. How is it used in Newton's method and Hessian methods? (2nd-order approximation)
Newton-style methods use not only first-order information (slope) but also second-order information (curvature). In second-order Taylor form, we approximate the local loss surface as a paraboloid and choose a step size that is more efficient for that shape.
A simple comparison:
- 1st order (gradient descent): strong at choosing direction,
- 2nd order (Newton): chooses both direction and step size more intelligently.
The Hessian is the key tool that stores curvature. It tells you which directions are steep and which are flat, so you can scale movement differently by direction. When conditions are good, Newton-type methods can converge much faster.
3. How is it used in numerical computing and deep learning practice?
In practice, we often reduce computation by approximating complicated functions on the interval we need, instead of evaluating them exactly every time. Taylor series is a core tool for that.
Typical uses include:
- approximating functions like exe^xex, sin⁡x\sin xsinx, and log⁡(1+x)\log(1+x)log(1+x) with polynomials on small ranges,
- simplifying activation or loss behavior on specific operating ranges,
- designing stable optimization steps from a local approximation of the current landscape.
In deep learning, global exact analysis is often less useful than understanding the neighborhood of the current parameters. The Taylor view makes computation lighter and improves interpretability of why updates move the way they do. So this is not only textbook math; it is practical reasoning used in real training pipelines.
The table below lists formulas and symbols for solving problems. See worked examples under the table for step-by-step solutions.
  • FormulaTn(x)=∑k=0nf(k)(a)k!(x−a)kT_n(x)=\sum_{k=0}^{n}\frac{f^{(k)}(a)}{k!}(x-a)^kTn​(x)=∑k=0n​k!f(k)(a)​(x−a)k
  • MeaningTnT_nTn​ = degree-nnn Taylor polynomial. f(k)(a)f^{(k)}(a)f(k)(a) = kkkth derivative at aaa. k!k!k! = factorial. (x−a)k(x-a)^k(x−a)k = distance from center raised to kkk.
  • FormulaT1(x)=f(a)+f′(a)(x−a)T_1(x)=f(a)+f^{\prime}(a)(x-a)T1​(x)=f(a)+f′(a)(x−a)
  • MeaningLinear approximation = tangent. Links to gradient steps in ML.
  • FormulaT2(x)=f(a)+f′(a)(x−a)+f′′(a)2(x−a)2T_2(x)=f(a)+f^{\prime}(a)(x-a)+\frac{f^{\prime\prime}(a)}{2}(x-a)^2T2​(x)=f(a)+f′(a)(x−a)+2f′′(a)​(x−a)2
  • MeaningQuadratic approximation. Links to Newton and Hessian.
  • Formulaa=0a=0a=0 (Maclaurin)
  • MeaningCenter at origin: Tn(x)=∑k=0nf(k)(0)k!xkT_n(x)=\sum_{k=0}^{n}\frac{f^{(k)}(0)}{k!}x^kTn​(x)=∑k=0n​k!f(k)(0)​xk.
  • FormulaTerm count at degree nnn
  • MeaningFrom x0x^0x0 to xnx^nxn → n+1n+1n+1 terms.
  • FormulaCoefficient of xnx^nxn (f(x)=c⋅xmf(x)=c\cdot x^mf(x)=c⋅xm)
  • MeaningIf n=mn=mn=m, coefficient is ccc; otherwise 0 (low-degree Maclaurin).
  • FormulaRemainder (concept)
  • MeaningAfter degree nnn, error is roughly order (x−a)n+1(x-a)^{n+1}(x−a)n+1.
  • FormulaML link
  • Meaning1st → SGD/gradient. 2nd → Newton/Hessian. Higher → numerical/function approximation.
FormulaMeaning
Tn(x)=∑k=0nf(k)(a)k!(x−a)kT_n(x)=\sum_{k=0}^{n}\frac{f^{(k)}(a)}{k!}(x-a)^kTn​(x)=∑k=0n​k!f(k)(a)​(x−a)kTnT_nTn​ = degree-nnn Taylor polynomial. f(k)(a)f^{(k)}(a)f(k)(a) = kkkth derivative at aaa. k!k!k! = factorial. (x−a)k(x-a)^k(x−a)k = distance from center raised to kkk.
T1(x)=f(a)+f′(a)(x−a)T_1(x)=f(a)+f^{\prime}(a)(x-a)T1​(x)=f(a)+f′(a)(x−a)Linear approximation = tangent. Links to gradient steps in ML.
T2(x)=f(a)+f′(a)(x−a)+f′′(a)2(x−a)2T_2(x)=f(a)+f^{\prime}(a)(x-a)+\frac{f^{\prime\prime}(a)}{2}(x-a)^2T2​(x)=f(a)+f′(a)(x−a)+2f′′(a)​(x−a)2Quadratic approximation. Links to Newton and Hessian.
a=0a=0a=0 (Maclaurin)Center at origin: Tn(x)=∑k=0nf(k)(0)k!xkT_n(x)=\sum_{k=0}^{n}\frac{f^{(k)}(0)}{k!}x^kTn​(x)=∑k=0n​k!f(k)(0)​xk.
Term count at degree nnnFrom x0x^0x0 to xnx^nxn → n+1n+1n+1 terms.
Coefficient of xnx^nxn (f(x)=c⋅xmf(x)=c\cdot x^mf(x)=c⋅xm)If n=mn=mn=m, coefficient is ccc; otherwise 0 (low-degree Maclaurin).
Remainder (concept)After degree nnn, error is roughly order (x−a)n+1(x-a)^{n+1}(x−a)n+1.
ML link1st → SGD/gradient. 2nd → Newton/Hessian. Higher → numerical/function approximation.

Worked examples

Example 1 — definition (True/False)
"A Maclaurin series is a Taylor series centered at a=0a=0a=0."
1) True
2) False
Maclaurin is exactly the special Taylor case with center 0. -> Answer 1
Example 2 — definitionChoice (multiple choice concept)
"Which formula is the correct first-order Taylor approximation?"
1) T1(x)=f(a)+f′(a)(x−a)T_1(x)=f(a)+f^{\prime}(a)(x-a)T1​(x)=f(a)+f′(a)(x−a)
2) T1(x)=f(a)+f′′(a)2(x−a)2T_1(x)=f(a)+\frac{f^{\prime\prime}(a)}{2}(x-a)^2T1​(x)=f(a)+2f′′(a)​(x−a)2
3) First-order approximation always equals the original function
First-order means tangent-line form, so 1) is correct. -> Answer 1
Example 3 — linearApprox (1st-order value)
"For f(x)=2x+1f(x)=2x+1f(x)=2x+1 and center a=0a=0a=0, what is T1(3)T_1(3)T1​(3)?"
For a linear function, first-order Taylor equals the function itself. 2⋅3+1=72\cdot3+1=72⋅3+1=7. -> Answer 7
Example 4 — quadraticApprox (2nd-order value)
"For f(x)=x2+2x+1f(x)=x^2+2x+1f(x)=x2+2x+1 and center a=0a=0a=0, what is T2(2)T_2(2)T2​(2)?"
This is already a quadratic polynomial, so the 2nd-order approximation matches exactly. 22+2⋅2+1=92^2+2\cdot2+1=922+2⋅2+1=9. -> Answer 9
Example 5 — maclaurinCoeff (read coefficient)
"In f(x)=5x3f(x)=5x^3f(x)=5x3, what is the coefficient of x3x^3x3 in the Maclaurin series?"
Just read the coefficient in front of x3x^3x3. -> Answer 5
Example 6 — derivativeAtCenter (derivative at center)
"For f(x)=x3f(x)=x^3f(x)=x3, what is f(3)(0)f^{(3)}(0)f(3)(0)?"
Differentiate x3x^3x3 three times to get 6. Constants stay 6 at x=0x=0x=0. -> Answer 6
Example 7 — termCount (number of terms)
"How many terms does a 4th-order Taylor polynomial have?"
From degree 0 to degree 4, so 4+1=54+1=54+1=5 terms. -> Answer 5
Example 8 — remainderOrder (order of remainder)
"After a degree-n=2n=2n=2 Taylor approximation, what is the typical order of the remainder term?"
1) (x−a)n(x-a)^n(x−a)n
2) (x−a)n+1(x-a)^{n+1}(x−a)n+1
3) (x−a)n+2(x-a)^{n+2}(x−a)n+2
The remainder typically starts one order higher. -> Answer 2
Example 9 — mlConcept (ML concept link)
"Which ML interpretation is closest to first-order Taylor approximation?"
1) One-step move based on current gradient
2) Always multiply by inverse Hessian
3) Defines batch normalization
First-order Taylor matches the gradient-based one-step intuition. -> Answer 1

문제

If the statement is true, choose 1; if false, choose 0.
The first-order Taylor polynomial equals the tangent line at aaa.
1 / 5