Softmax (Turn into Probabilities)

Softmax is a function that converts multiple scores (numbers) into probabilities. All values become between 0 and 1, and they sum to exactly 1. So you can read them as probabilities.

The formula is $$\mathrm{softmax}(x_i) = \frac{e^{x_i}}{\sum_{j} e^{x_j}}$$. Because it uses powers of e (≈2.718), the largest score gets amplified significantly while others shrink relatively. The gap between 1st and 2nd place becomes more pronounced.

Example: scores [3, 1, 0] → e³≈27, e¹≈2.7, e⁰=1 → sum ≈ 23.7 → probabilities → [0.84, 0.11, 0.04]. The score of 3 was only 3× larger than 1, but the probability is about 8× larger!

Softmax is used at the final layer of almost every classification model. 'This photo is 70% dog, 25% cat, 5% bird' lets you see per-class probabilities and how confident the model is.

When combined with cross-entropy loss during training, the gradients work out cleanly and stably. The model naturally learns to 'increase the correct class probability and decrease the rest.'

Softmax's property of 'all positive values that sum to 1' exactly matches the definition of a probability distribution. This makes it the most natural way to convert scores to probabilities, both statistically and theoretically.

Image classification: The model's final layer outputs scores (logits) like [5.2, 2.1, 0.8, ...]. Softmax converts them to [0.70, 0.25, 0.05, ...]—probabilities for each class. The highest probability class is the final answer.

Chatbots & translators: When ChatGPT picks the next word, it scores every word in its vocabulary (tens of thousands!), converts to probabilities via softmax, and samples a word based on those probabilities. High-probability words appear often, but occasionally low-probability words are picked for variety.

Attention mechanism: In translators, relevance scores for 'which input words to focus on' are passed through softmax to become probabilities (weights). These weights create a weighted average of inputs that emphasizes the most relevant parts.

Computation order: ① Compute $Z = W \cdot X + b$ (logits) ② Compute $e^Z$ (problem uses $e \approx 3$) ③ Compute $\Sigma$ (sum) = add all $e^Z$ values ④ $Y = \frac{e^Z}{\Sigma}$ (divide each by the sum). Follow this order.

Finding blanks: If Y is blank, compute 'that $e^Z \div \Sigma$.' If $e^Z$ is blank, compute '$Y \times \Sigma$.' If Z is blank, reverse from $e^Z$. If $\Sigma$ is blank, just add all $e^Z$ values.

Verification: After computing, check that all Y values are between 0 and 1 and sum to 1. If not, there's a calculation error. Also confirm whether the problem uses $e \approx 3$ or $e \approx 2.718$.