Ch.04

Loss Functions: Class Imbalance and Metric Learning

Imagine a school exam with 100 questions: 99 easy arithmetic problems and just one very hard essay question. Get all the arithmetic right and you score 99 — it feels like you're doing great. AI faces the same trap with class imbalance : follow the majority and you can hit 99% accuracy while missing every rare but critical case (rare disease, defective parts, fraud). This chapter redesigns the loss function itself to handle imbalance, then goes further into metric learning — teaching similarity through distance between data points, not only class labels. Study weighted CE, Focal loss, and Triplet/Contrastive loss in concept \to intuitive analogy \to formula \to practice tips order, and avoid the trap of "high score, wrong priorities."

The 2\times2 diagram below shows how four losses reshape embedding space before \to after training.

Each panel shows before (left) → after (right). Contrastive pulls positive pairs and pushes negatives apart. Triplet pulls anchor–positive close and keeps negative beyond the margin. Weighted CE adds w↑ for minorities; Focal shrinks easy dots and enlarges hard ones.

Pull similar, push different — learning by distance

Imbalance: weights & Focal; similarity: Contrastive & Triplet

Loss Functions Deep Dive: Learning from Imbalance and Distance

1. What is class imbalance? (AI trapped by majority vote)

Concept: Some classes have far more samples than others. With plain training, the model focuses on the majority class because it is easiest to get right.

Intuitive analogy: In defect detection, 99% of items are fine. Predict "all normal" and accuracy is 99% — but the real goal is finding the hidden 1% defects (minority class).

Practice tip: Add per-class weights or Focal loss so the model pays attention to rare, important cases.

2. Weighted cross-entropy (Weighted CE)

Concept: Multiply each class

c

by a weight

w_c

. Misclassifying a minority class costs more. Core formula:

L = - w_c \log(p_c)

Intuitive analogy: Change the grading rubric: 99 easy questions worth 1 point each, one rare essay worth 100 points — the student (AI) cannot ignore the essay.

Practice tip: Weights that are too large can overfit noisy minority samples. Tune gradually and watch per-class metrics.

3. Focal loss — skip what you know, focus on what you don't

Concept: Easy samples (high predicted probability

p_t

) get much smaller loss; hard samples dominate training. Core formula:

L_{\text{focal}} = - (1-p_t)^\gamma \log(p_t)

, where

\gamma

controls focus.

Intuitive analogy: Efficient cramming: skip chapters you already ace; pour time into weak topics you keep missing.

Practice tip: Very effective when background (majority) vs object (minority) ratios are extreme, e.g. object detection.

4. Metric learning — cluster friends, separate strangers

Concept: Instead of only picking a class label, the model learns distance — cat photos near cats, far from dogs. Triplet loss uses anchor, positive, and negative. Core formula:

L = \max(0, d(a,p) - d(a,n) + \alpha)

Intuitive analogy: Wedding seating: close friends (positive) at the same table; awkward pairs (negative) kept at least safety distance

\alpha

apart.

Practice tip: Widely used for face recognition, similar-item search, and any task comparing similarity between examples.

Losses at a glance

Weighted CE — penalizes minority misclassification more heavily.

Core formula:

L = -w_c \log(p_c)

Symbols —

p_c

is the model's predicted probability for the true class $c$ (0–1).

w_c

is the point value for class

c

; rare classes get larger weights.

\log(p_c)

grows when the model is wrong (low

p_c

), and the leading minus makes training raise

p_c

Weight rule: with

N

total samples and

K

classes,

w_c \propto N/(K n_c)

— smaller class count

n_c

→ larger

w_c

Numeric example:

p_c=0.2

w_c=5

gives loss

\approx -5\log(0.2) \approx 8

— the same mistake hurts more when weighted.

Analogy: plain cross-entropy with per-class grading weights.

Focal loss — down-weights easy samples, focuses on hard ones.

Core formula:

L_{\text{focal}} = -(1-p_t)^\gamma \log(p_t)

Symbols —

p_t

is the model's confidence on this sample (probability of the true label).

(1-p_t)

measures uncertainty;

(1-p_t)^\gamma

shrinks easy samples (

p_t

high) more when

\gamma

is large.

\log(p_t)

is the same CE backbone.

Numeric example:

p_t=0.9

\gamma=2

→

(1-0.9)^2=0.01

— loss drops to ~1%, so mastered samples barely matter. At

p_t=0.3

(0.7)^2=0.49

— still a strong penalty.

Analogy: skip chapters you already ace; drill the ones you keep missing.

Triplet loss — pull friends close, push strangers apart.

Core formula:

L = \max(0,\, d(a,p) - d(a,n) + \alpha)

Symbols —

a

is the anchor,

p

a positive (same identity),

n

a negative (different identity).

d(\cdot,\cdot)

is distance (e.g. L2).

\alpha

is the margin: negative must stay at least this far from the anchor.

\max(0,\cdot)

means no penalty once the layout is already good enough.

Numeric example:

d(a,p)=1

d(a,n)=4

\alpha=0.5

→

1-4+0.5=-2.5

→

\max(0,-2.5)=0

(negative far enough). If

d(a,n)=1.2

→

1-1.2+0.5=0.3

— still too close, 0.3 penalty.

Analogy: wedding seating — friend (

p

) beside anchor, awkward guest (

n

) at least

\alpha

away.

Contrastive loss — pull same identity (or augmentations) together, push different identities apart.

Core formula — positive pair:

L_+ = 0.5\,d^2

· negative pair:

L_- = 0.5\,\max(0,\, m-d)^2

Symbols —

d

is distance between two embeddings. For positives, smaller

d

→ smaller loss (pull). For negatives,

m

is the margin; when

d \ge m

\max(0,m-d)=0

so no further push is needed. The

0.5

is a scaling constant.

Numeric example: positive with

d=0.4

→

0.5\times0.16=0.08

(close = low cost). Negative with

d=0.6

m=1

→

0.5\times(0.4)^2=0.08

— still too close, keep pushing.

Analogy: cluster photos of the same person; keep different people at least

m

apart.

Why it matters

1. Filter out fake 100% report cards

99% accuracy on imbalanced data can be meaningless. To see whether the model solves what matters, encode priorities in the loss with weighted CE or Focal loss — not accuracy alone.

2. The loss sets your priorities

What the model should care about most is encoded in the shape of the loss. With the wrong objective, more training rarely fixes the real problem. For imbalance and distance learning, the key is telling the model where penalties should bite.

3. Backbone of search and recommendation

When you search "similar clothes" or unlock your phone with your face, the model compares how alike two images are (distance) — not multiple-choice labels. Metric learning powers these services.

How it is used

① Fix imbalance — diagnose in order

Symptom: Model nails common classes, fails rare ones.

Steps:

1. Check distribution — how skewed are class counts?

2. Swap the loss — try weighted CE or Focal loss.

3. Review training settings — if the loss changed, revisit learning rate and batch size too.

② Metric learning — train in pairs/triplets

Metric learning needs data prepared as sets:

- Triplet: (reference photo, same person, different person).

- Contrastive: (original, augmented original) as positive — pull together; push others away.

The model learns distances so similar items cluster.

③ Drop easy problems (Hard Negative Mining)

Training only on apple vs car — too easy, learning stalls. Find hard negatives that look similar but differ — like tough mock exams before the real test.

④ Pick your ruler — L2 vs cosine

- L2 distance: straight-line gap; cares about magnitude and position.

- Cosine similarity: angle between arrows; good for meaning or "vibe" of embeddings.

Quick fixes: ignores minority → weights/Focal; only easy samples → raise Focal $\gamma$ ; poor search → add hard negatives.

Summary

One line: This chapter showed how to design loss functions for class imbalance and how to learn similarity through embedding distance.

The four key losses work as follows. Weighted CE uses inverse-frequency

w_c

to punish minority mistakes more heavily. Focal loss combines

\alpha

with

(1-p_t)^\gamma

to shrink the influence of easy samples. Triplet loss uses margin

\alpha

to keep positives near the anchor and negatives farther away. Contrastive loss organizes embedding space by pulling positive pairs together and pushing negatives apart.

In practice, if the model only predicts the majority class, switch to weighted CE or Focal and track per-class F1. If training stalls on easy samples, tune Focal

\gamma

— but too large a value can destabilize learning. If Triplet loss stays near zero, negatives may be too easy and hard negative mining is often needed. For face authentication or similarity search, metric embeddings and the choice between L2 and cosine distance matter most.

When tuning, start by checking class distribution and metrics, then pick a loss and adjust learning rate and related settings. Change one thing at a time so you can see what actually helped.

Problem-solving guide

Problems in this chapter fall into two lines: imbalanced classification and distance-based similarity learning. With imbalance, high accuracy can hide the fact that rare classes are missed — plain cross-entropy often rewards the majority. Weighted CE assigns

w_c

per class to penalize minority mistakes more; Focal loss uses

(1-p_t)^\gamma

to down-weight easy samples so hard ones drive training. Metric learning, by contrast, teaches distance in embedding space rather than a single label. Triplet loss pulls anchor–positive together and pushes negative past margin

\alpha

via

L=\max(0,d(a,p)-d(a,n)+\alpha)

; Contrastive loss pulls positive pairs and pushes negatives apart. If the stem says the model only gets the majority right, think weights/Focal; if negatives are too easy, think hard mining; for face auth or search, think metric embeddings first.

Definition problems — ask what the loss penalizes most. For "Why give a large

w_c

to the minority in weighted CE?", option

① skips backprop (unrelated) and

③ fixes the batch (not the point). Weighted CE raises the cost of misclassifying the minority → Answer 2.

Scenario problems — read the data setup first. With 99% normal and 1% fraud, keeping plain CE only (①) or deleting the minority (③) misses the goal. The first loss to try is weighted CE or Focal (②). → Answer 2

Calculation problems — plug into the formula step by step. With

w_B=N/(K\cdot n_B)

N=1000

K=2

n_B=100

1000/(2\cdot100)=5

. → Answer

②

Definition example — "What does

(1-p_t)^\gamma

do in Focal loss?" It down-weights easy samples so training focuses on hard ones — not

① equal loss or

③ raising LR. → Answer 2

True/False example — "Triplet loss needs anchor, positive, and negative." True — it learns from three-point distance relations. → Answer 1

Application example — "Train embeddings for face verification" → Triplet/Contrastive distance learning fits better than plain classification. → Answer 1

Choice example — With extreme imbalance (e.g. detection backgrounds), Focal (②) is often preferred over CE only (①). → Answer 2

Concept example — With

d(a,p)=1

d(a,n)=4

\alpha=0.5

\max(0,1-4+0.5)=0

— margin already satisfied, no further penalty. → Answer 3

Calculation example — L2 distance

(0,0)

–

(3,4)

\sqrt{9+16}=5

. → Answer

②

Loss Functions Deep Dive: Learning from Imbalance and Distance

1. What is class imbalance? (AI trapped by majority vote)

Concept: Some classes have far more samples than others. With plain training, the model focuses on the majority class because it is easiest to get right.

Intuitive analogy: In defect detection, 99% of items are fine. Predict "all normal" and accuracy is 99% — but the real goal is finding the hidden 1% defects (minority class).

Practice tip: Add per-class weights or Focal loss so the model pays attention to rare, important cases.

2. Weighted cross-entropy (Weighted CE)

Concept: Multiply each class

c

by a weight

w_c

. Misclassifying a minority class costs more. Core formula:

L = - w_c \log(p_c)

Intuitive analogy: Change the grading rubric: 99 easy questions worth 1 point each, one rare essay worth 100 points — the student (AI) cannot ignore the essay.

Practice tip: Weights that are too large can overfit noisy minority samples. Tune gradually and watch per-class metrics.

3. Focal loss — skip what you know, focus on what you don't

Concept: Easy samples (high predicted probability

p_t

) get much smaller loss; hard samples dominate training. Core formula:

L_{\text{focal}} = - (1-p_t)^\gamma \log(p_t)

, where

\gamma

controls focus.

Intuitive analogy: Efficient cramming: skip chapters you already ace; pour time into weak topics you keep missing.

Practice tip: Very effective when background (majority) vs object (minority) ratios are extreme, e.g. object detection.

4. Metric learning — cluster friends, separate strangers

Concept: Instead of only picking a class label, the model learns distance — cat photos near cats, far from dogs. Triplet loss uses anchor, positive, and negative. Core formula:

L = \max(0, d(a,p) - d(a,n) + \alpha)

Intuitive analogy: Wedding seating: close friends (positive) at the same table; awkward pairs (negative) kept at least safety distance

\alpha

apart.

Practice tip: Widely used for face recognition, similar-item search, and any task comparing similarity between examples.

w_c

per class to penalize minority mistakes more; Focal loss uses

(1-p_t)^\gamma

\alpha

via

L=\max(0,d(a,p)-d(a,n)+\alpha)

Definition problems — ask what the loss penalizes most. For "Why give a large

w_c

to the minority in weighted CE?", option

① skips backprop (unrelated) and

③ fixes the batch (not the point). Weighted CE raises the cost of misclassifying the minority → Answer 2.

Calculation problems — plug into the formula step by step. With

w_B=N/(K\cdot n_B)

N=1000

K=2

n_B=100

1000/(2\cdot100)=5

. → Answer

②

Definition example — "What does

(1-p_t)^\gamma

do in Focal loss?" It down-weights easy samples so training focuses on hard ones — not

① equal loss or

③ raising LR. → Answer 2

True/False example — "Triplet loss needs anchor, positive, and negative." True — it learns from three-point distance relations. → Answer 1

Application example — "Train embeddings for face verification" → Triplet/Contrastive distance learning fits better than plain classification. → Answer 1

Choice example — With extreme imbalance (e.g. detection backgrounds), Focal (②) is often preferred over CE only (①). → Answer 2

Concept example — With

d(a,p)=1

d(a,n)=4

\alpha=0.5

\max(0,1-4+0.5)=0

— margin already satisfied, no further penalty. → Answer 3

Calculation example — L2 distance

(0,0)

–

(3,4)

\sqrt{9+16}=5

. → Answer

②