Ch.06
Swin Transformer: Hierarchical Windows and Global Context
Swin Transformer
Patch Merging
Patches are small tiles; windows set how far attention looks at once. W keeps attention inside the window; SW shifts the window so neighbors briefly connect. Merging halves height and width while stacking deeper features.
Split a grid into four checkerboard-like groups, then concatenate matching positions to get temporary (preserve information first). After that, apply LayerNorm → Linear to compress key features . In short: spatial size shrinks, semantic channel depth is retained via compressed stacking.
The ViT (Vision Transformer) from Chapter 05 is like a giant town-hall debate where all image pixels discuss everything at once (global attention). Accuracy is strong, but as participants (resolution) increase, memory cost explodes quadratically ()—a critical weakness.
Swin Transformer (Shifted Window Transformer) solves this elegantly. First, it lets tokens talk only inside small meeting rooms (windows). In the next layer, it slightly moves the partitions (shift) so neighboring groups can exchange information. Then it gradually compresses four workers into one manager (Patch Merging) to build a bigger-picture view. In this chapter, we study how this divide-and-conquer intuition cuts computation so dramatically in math terms.
How to read the key formulas (Swin)
1) W-MSA (small-room rule): If window side length is , attention inside one window handles roughly relations. Instead of one global meeting, computation is split into many small rooms.
2) SW-MSA (moving partitions): In the next block, shift windows by and keep only valid query-key links with an attention mask. Neighboring windows can exchange information while invalid links are blocked.
3) Patch Merging (information-preservation view):
① Concatenate four patches (each with channels) at one location, giving temporary (stack first, do not discard).
② Then Linear compresses key information to .
So resolution drops to while channel depth increases to reduce information loss.
4) Why resolution↓ and channels↑? (deeper intuition):
① Information-capacity view
A rough capacity proxy is . If spatial size becomes 1/4 and channels stay fixed, representational capacity drops too sharply, and fine detail is lost. Increasing channels () buffers this loss.
② Compute-vs-expression balance
Keeping high resolution is expensive; blindly increasing channels is also expensive. Swin reduces token count first (memory/speed gain), then deepens per-token semantics via channels.
③ Context/receptive-field view
Deeper stages should model larger structures and long-range relations. Lower resolution expands effective context; more channels provide room to encode richer semantic factors.
④ Practical one-liner
Only reducing resolution causes "amnesia," only increasing channels causes "compute explosion." Swin uses resolution↓ + channels↑ as an engineering trade-off to control both.
Swin Transformer: where intuition meets equations
1. Window Attention (W-MSA): putting computation on a diet in small rooms
Metaphor: If tens of thousands of people all talk at once (ViT), it's chaos. Instead, split the image into 7x7-size meeting rooms (windows), and let attention happen only among tokens in the same room.
Core equation: Let total patch count be and window side length be . ViT cost grows as . W-MSA becomes . Since is a fixed small constant (often 7), overall cost scales almost linearly with , i.e., practical behavior.
2. Shifted Window (SW-MSA): twist partitions to communicate
Metaphor: If everyone talks only to their own room forever, they become siloed. So in the next layer, partitions are shifted by half a window (down-right). Patches that were in neighboring rooms now share one room and exchange information.
Core equation: Mathematically, windows are shifted by patches. Boundary fragments are handled cleanly with cyclic shift plus an attention mask, so invalid interactions are blocked without large extra memory.
3. Patch Merging: four workers become one manager
Metaphor: As depth increases, fine-grained local opinions are summarized into broader context. Swin groups neighboring patches (4 total) into one unit.
Core equation: Resolution halves to , while channel width temporarily becomes . A Linear layer compresses it to . So across stages, spatial size shrinks while semantic depth grows—a multi-scale pyramid.
4. Two-Successive Blocks: always run as a pair
Concept: Swin blocks work in 2-step combos: (regular window → shifted window).
* : step 1, talk inside your room
* : step 2, shift partitions and talk across neighbors
Why it matters
An engineering win over heavy math
With ViT, once resolution rises (e.g., beyond 256x256), quadratic attention quickly becomes impractical on normal GPUs. Swin constrains this to near-linear scaling, opening the door to training 4K driving video and giga-pixel pathology slides (WSI) without constant OOM failures.
A paradigm shift for dense prediction
Image classification can already work with ViT. But object detection and semantic segmentation require robust multi-scale pyramids because object sizes vary greatly. Swin brings this pyramid behavior into transformers through patch merging, replacing CNN backbones (e.g., ResNet) in many SOTA systems.
Toward a unified visual backbone
Instead of "ViT for classification, ResNet for detection," Swin enabled one transformer-family backbone to serve classification, detection, and segmentation together, reducing maintenance and research overhead in production.
How it is used
1. Debugging rule #1: keep input size a multiple of 32
The most common real-world Swin error is shape mismatch. Patch merging is repeatedly applied (), so resolution keeps halving: . Input height/width should therefore be divisible by 32 (e.g., 224, 256, 512, 1024). Otherwise add zero-padding or resize first.
2. In practice, start from fine-tuning—almost always
Training Swin from scratch on a small in-house dataset is usually inefficient and prone to overfitting. Standard workflow: load pretrained backbones (Swin-T, Swin-B) from Hugging Face/MMDetection, replace only the task head, and fine-tune.
3. OOM survival guide
Even with near-linear complexity, Swin is still heavy. If memory crashes:
1) reduce batch size first,
2) lower input resolution (e.g., 1024 → 512),
3) reduce window size (e.g., 7 → 5).
Bonus: `gradient_checkpointing=True` can save substantial VRAM at a speed cost.
Summary
💡 [Core cheat sheet]
* Operating philosophy: split what can be split (W-MSA), reconnect what is disconnected by shifting partitions (SW-MSA), and aggregate four patches into one as depth grows (Patch Merging).
* Why it matters in vision AI: Swin cures the transformer's high-resolution Achilles heel (from toward practical ) while inheriting CNN-style multi-scale pyramids.
* Production mindset: "Use input sizes divisible by 32, start from pretrained weights, and tune window size plus batch when memory is tight."
Notes for problem solving
Common Swin problem-solving frame (check these 3 lines first)
1) Is this asking about W-MSA / SW-MSA / Patch Merging?
2) If it is a calculation, identify units first: token count , window size , channel , resolution .
3) If merging appears, apply the default transform immediately:
* Spatial:
* Channel:
Example (Concept)
"Which is NOT core to Swin?" → fixed full global attention every layer is not core.
Example (True/False)
"Patch merging increases spatial resolution." → false. Answer 0
Example (Scenario)
"High-resolution OOM occurs. What to try first?" → check batch↓, resolution↓, window size()↓ in order.
Quick examples by problem type (same layout as Ch05, Swin-specific content)
Example (Multiple-choice calculation)
"If , what is global attention ?" → .
Example (Cumulative merging)
"Resolution after 3 patch-merging operations?" → each step halves side length, so : .
Example (Configuration/counting)
"Square grid with 8 windows per side: total windows?" → .
Example (Integrated reasoning)
"Input , patch , tokens after one merging?"
1) Initial tokens:
2) After one merge:
→ Answer 784
One practical tip
For calculation problems, first separate what becomes 1/2 and what becomes 2x. (Spatial shrinks, channel depth grows.)