I aM Doo AI

Ch.07

Ensemble and Random Forest: The Wisdom of the Crowd

Ensemble methods combine predictions from multiple models to produce a single, often better prediction. This chapter explains bagging, boosting, stacking, and random forest—where many decision trees vote or average—so beginners can follow the idea of collective intelligence.

ML diagram by chapter

Select a chapter to see its diagram below. View the machine learning flow at a glance.

Combine predictions from multiple models (trees) by voting or averaging to get the final prediction.

  • ① Draw bootstrap samples from training data and train multiple trees
  • ② Each tree predicts independently
  • ③ Classification: majority vote; Regression: average → final prediction
  • ④ The final prediction is determined
DataTree 1Tree 2Tree 3Vote/AveragePrediction

Ensemble and Random Forest: The Wisdom of the Crowd

The core idea of ensemble: many hands make light work — An ensemble builds a team of multiple models and combines their predictions to reach a final answer. Like a jury voting on a verdict, using many models instead of one sharply reduces the chance of wrong answers (variance) and makes predictions more stable. For classification we use majority vote; for regression we use the average of predictions.
Why are many better than one? (Wisdom of the crowd) — If you ask 100 people to guess a cow's weight, individual guesses may be off, but the average of 100 guesses is often surprisingly close to the true weight. When models independently judge and we combine results, their random errors tend to cancel out and the shared signal remains.
Three main ensemble methods: Bagging, Boosting, Stacking —
(1) Bagging: Each model gets a different random subset of data (like different practice tests); then they vote.
(2) Boosting: The next model focuses on what the previous one got wrong, learning sequentially from mistakes.
(3) Stacking: A meta-model takes the reports of base models and makes the final decision.
Random Forest: a forest of diverse trees — Bagging with decision trees: grow hundreds of trees. To keep them diverse, each tree is trained on a random subset of features at each split. Some trees rely on "age", others on "income", maximizing diversity.
Voting and averaging in a formula — For classification, majority vote means "the class that most trees chose". For regression (e.g. house price), average all tree predictions: y^=1B∑b=1By^b\hat{y} = \frac{1}{B}\sum_{b=1}^B \hat{y}_b where BB is the number of trees and y^b\hat{y}_b is the bb-th tree's prediction. (e.g. three trees predict 100, 150, 200 → final prediction 150)
OOB (Out-of-Bag) evaluation — In bagging/random forest, each tree is trained on a random sample of the data. The left-out samples (Out-of-Bag) can be used to evaluate those trees that did not see them—like a built-in validation set without holding out separate test data.
A stable forest that doesn't sway — A single decision tree can change a lot when data changes slightly. A forest of hundreds of trees stays stable; a few wrong trees don't change the overall vote. This leads to strong, reliable performance in practice.
Natural extension of Ch06 Decision Tree — The same tree structure (impurity, information gain) is reused. You're not learning new rules—just how to combine many trees with voting, so the previous chapter's knowledge is fully used.
The go-to model in industry and competitions — Random forest often works very well with little tuning, so it's many practitioners' first choice. It also provides feature importance, which helps explain which variables matter most.
General-purpose for business (classification and regression) — From "Is this email spam?" to "What will tomorrow's stock price be?", ensembles are used across almost every business problem.
Finding what matters (feature importance) — If trees in a loan model rely most on "income", that variable is the most important for the decision. This helps filter out unnecessary data.
Wide real-world use — Fraud detection, recommendation systems (e.g. Netflix, YouTube), equipment failure prediction—wherever accuracy and stability matter.
Ensemble / Random forest — solving guide —
(1) Majority vote: Compare votes for class 0 vs class 1; the majority is the final prediction (0 or 1).
(2) Vote count: The number of votes for the winning class.
(3) Regression mean: Sum of tree predictions divided by number of trees; round if needed.
(4) OOB: Number of trees whose bootstrap sample did not include this sample.
(5) Formula: y^=1B∑b=1By^b\hat{y} = \frac{1}{B}\sum_{b=1}^B \hat{y}_b — BB is number of trees, y^b\hat{y}_b the bb-th prediction. Divide sum by BB for the mean. See the Explanation for solving the problems table below.