Ch.01
Data Scaling and Distribution Transformation
Human age might range from 20 to 70, while annual income can swing from 20M to 200M KRW. Machine learning does not understand "years" or "won"—it only sees how big the number looks. Feed the raw table as-is and the model may think: "Income is 50 million, so age 30 is basically noise I can ignore."
Data scaling converts elephant-sized and mouse-sized features onto the same percentage or score sheet so they compete on fair footing. This chapter explains standardization (like SAT-style z-scores), Min-Max (squeeze into a tiny box), and robust scaling (ignore the opera singer who wandered into karaoke night).
Intermediate ML diagram by chapter
Select a chapter to see its diagram below. View the intermediate ML flow at a glance.
Data Scaling: Matching Units and Handling Distributions
1. What is scaling? (One height chart for every feature)
Imagine a dating-app matcher comparing age gap vs income gap. A 5-year age difference and a 1M KRW income gap feel similar to humans—but the computer sees "1,000,000 vs 5" and treats age like dust. Scaling puts every feature on a fair ruler (often 0–1 or mean 0) so magnitude stops bullying meaning.
2. Standardization: SAT-style z-scores
Formula:
You scored 90 on an easy language test and 80 on a brutal math test. Raw numbers favor language—but after adjusting for mean () and spread (), the hard-test score can win. Standardization gathers features near mean 0 and variance 1 on a shared bell-curve field.
3. Min-Max: squeeze into a miniature box
Formula:
Last place becomes , first place becomes , everyone else lands at 0.2, 0.5, 0.8 in between. Even million-scale raw values get pressed into the tight box—great when you need a fixed input range (e.g. pixels).
4. Robust scaling: ignore the opera star at karaoke
Formula:
One world-class singer joins neighborhood karaoke and wrecks the average skill score—everyone looks tone-deaf. Robust scaling uses the median (middle person) and IQR (normal middle 50%) so extreme outliers like Pavarotti cannot rewrite the ruler.
5. Scaling vs the word "normalization"
The same word can mean different things by context. Normalization sometimes means Min-Max scaling, and in deep learning it can mean L2 weight regularization. This chapter is feature preprocessing (matching units and ranges)—not basic ML Ch.13 regularization.
Distance skew shows up in the numbers. With a 5-year age gap and a 1M KRW income gap, Euclidean distance is about —almost all from income. After standardization both axes weigh more evenly so distance/margin models like SVM can look at age and income together.
Scaling methods compared
Why each method behaves as it does, and which data it suits—explained in prose.
| Method | Why it works this way |
|---|---|
| Standardization | With you ask how far a point sits from the mean in standard-deviation units. Most values cluster near 0 with spread near 1, which helps distance/margin models like SVM compare features on similar footing. But and use every point, so one outlier can shift the mean and inflate , pushing most data to odd scores. |
| Min-Max scaling | pins the minimum at and maximum at , scaling everything else by rank in between. Output is always , which fits pixels (0–255 → 0–1) and neural nets that expect a fixed input box. The catch: the ruler depends on only min and max, so one extreme can stretch and squash everyone else into 0.01–0.02. |
| Robust scaling | uses the median and IQR (middle 50% width) instead of mean and standard deviation. Extremes barely move median or IQR, so for long-tailed or outlier-heavy data like income or payments, the ruler stays anchored to the typical middle bulk. |
Why it matters
1. Oxygen for distance and margin models (SVM, etc.)
These models rely on literal distance or margins. Without scaling it is like plotting a map with X in centimeters and Y in kilometers.
Picture a table with only age (~30–40) and income (~4,500–6,100 in ten-thousands KRW). The income bars swallow the chart. The left panel below is that range mismatch; the right shows z-scores with balanced weight.
2. Training speed booster (hot-dog hill vs rice-bowl valley)
Unscaled loss landscapes look like long hot-dog hills: gradient descent zigzags forever. After scaling the bowl becomes round and smooth—you slide to the bottom much faster.
3. No cheating—data leakage warning
Computing the exam average from tomorrow's answer key before you sit the test is cheating. Setting scaling rules using train and test together leaks future information. Models ace practice and fail production.
How it is used
Choose the right scaler for the situation
No single scaler suits every dataset. Look at the shape of your data first.
If outliers or extremes have wrecked the mean, Robust often beats standardization that relies on mean and standard deviation. Median and IQR set the ruler so one outlier like Pavarotti cannot rewrite the whole scale.
If values already live in a fixed box—pixels from 0 to 255—and you need inputs in something like , Min-Max is a natural fit. Minimum becomes , maximum becomes , everyone else lands in between.
For general numeric data or deep learning when you are unsure, teams often try standardization first. The table below compares all three formulas and traits.
Apply the same ruler to train, validation, and test
In practice you split data into train, validation, and test. Scaling rules—mean, min, max, and so on—are learned from training data only, then applied unchanged to validation and test. Recomputing those rules with validation or test mixed in is like grading tomorrow's exam with today's mock scores: data leakage.
The same rule holds in cross-validation. For each fold, set the ruler from that fold's training rows only and apply it to the validation rows. Even when you repeat mock exams to estimate performance, the grading key must come from training data each time, or scores will look better than they are in production.
Log-transform long tails before scaling
YouTube views or GDP often have fat right tails. Before scaling, try to flatten the shape, then standardize—a common combo in practice.
Model-by-model exceptions
Some models almost require scaling; others can skip it. Algorithms that use distance or margins directly—such as SVM—get pulled toward whichever feature has the biggest numbers if rulers do not match, and performance often collapses when you omit scaling.
Tree models and random forests only ask whether 30 is greater than 3. They care about rank, not whether income is stored as 45 million or 4,500, so you can usually leave features unscaled.
Neural nets and other gradient-based learners nudge weights in tiny steps. When input ranges differ wildly, optimization zigzags and slows. Min-Max or standardization flattens the loss landscape and keeps training steadier.
In practice
One-liner: Scaling puts elephant-sized and mouse-sized features on a fair ruler so distance/gradient models are not pulled by one axis.
In practice you can choose quickly. Heavy outliers or long tails → try Robust or log then standardize. Pixels or fixed inputs → Min-Max is handy. SVM, neural nets on ordinary numeric features → standardize first. Trees and random forests care about rank, so scaling is usually optional.
One rule is non-negotiable: never relearn scaling rules on test or validation—set the ruler on training (or each CV train fold) only, then apply it everywhere else.