Phase 2: Classical Machine Learning

Decision Trees & Random Forests

Intermediate ~3 min read
Think of it this way A friendly analogy. Read this if the technical version feels dense. Show Hide

Imagine you’re a super chef trying to figure out the best way to bake the perfect batch of cookies, no matter what ingredients you have in your pantry. A "Decision Tree" is like a very special recipe guide that helps you make choices step-by-step. It asks a series of simple questions: "Do you have chocolate chips?" If yes, turn to page 5. If no, turn to page 10. Each question helps you narrow down the best path, leading you to the exact ingredients and steps for a specific type of cookie. It’s a bit like a flowchart for cooking, where every decision brings you closer to your delicious goal, making sure you use what you have to get the best result.

The challenge with one of these super-specific recipe guides is that it can sometimes be too precise. If your cookie recipe only works perfectly when you have exactly 3.2 chocolate chips, precisely this brand of flour, and bake in that specific oven, it might make amazing cookies once. But if you change kitchens, use a slightly different kind of flour, or have slightly fewer chocolate chips, the recipe might suddenly not work well at all. It’s like a recipe that's too picky and only works in one very specific situation. It doesn't "generalize" well to other kitchens or ingredients, making your cookies turn out less than perfect.

That's where "Random Forests" come in! Instead of relying on just one super-picky cookie recipe guide, imagine you have a whole forest of many different recipe guides, all slightly unique. Each one might have learned to bake cookies a little differently – maybe one focuses on crunchy cookies, another on chewy ones, and another on nut-free options. To make the absolute best final decision for your cookies, you bake a tiny test batch with a few different guides, and then you ask all the guides for their opinion. If most of them say, "add more sugar," or "bake for 12 minutes," you go with what the majority of your expert recipe guides recommend.

By combining the wisdom of all these different "recipe experts," your final cookie decision is much more reliable and delicious, even if you change up ingredients a bit or bake in a different oven. This same idea helps computers make really smart predictions in the real world. For example, it can help decide if an email is spam (by looking at many different rules), or predict if a certain plant will grow well in a garden (by considering many different factors like soil and sun). This means you can help computers make more accurate and trustworthy decisions for all sorts of cool things, from sorting your photos to helping doctors understand health patterns!

Decision Trees are a fundamental supervised learning algorithm that models decisions in a tree-like structure, much like a flowchart. They work by recursively splitting the dataset into subsets based on the values of input features. At each "node" in the tree, the algorithm selects the feature and threshold that best separate the data, aiming to make the resulting subsets as "pure" as possible (meaning they contain mostly one class in classification, or have a low variance in regression). This process continues until a stopping criterion is met, like reaching a maximum depth or having too few samples in a node. Decision Trees are highly interpretable, allowing you to visually trace the decision path for a prediction, but a single deep tree can easily overfit to the training data, leading to poor generalization on unseen examples.

To address the overfitting issue and improve predictive power, we turn to Random Forests. A Random Forest is an ensemble learning method that builds a multitude of individual Decision Trees during training. Instead of relying on a single tree, it combines the predictions of many trees to make a final, more robust prediction. Each tree in the forest is trained on a different random subset of the training data (bootstrap aggregating or "bagging") and, crucially, considers only a random subset of features at each split point. For classification, the final prediction is determined by a majority vote among the trees; for regression, it's the average of their predictions. This "wisdom of the crowd" approach significantly reduces variance, improves generalization, and makes Random Forests far more robust to noise and overfitting than a single Decision Tree.

When choosing between the two, consider your priorities. If interpretability and explainability are paramount, a simpler Decision Tree (perhaps with limited depth) might be preferred. However, for most real-world applications where predictive accuracy is key, Random Forests are generally the go-to choice due to their superior performance and robustness. While individual trees in a Random Forest are less interpretable, the overall model's feature importance can still be analyzed, giving insights into which features contribute most to the predictions. Both algorithms are versatile and can handle various data types, making them powerful tools in an ML engineer's toolkit.

Key Takeaways

  • Decision Trees: Are interpretable, flowchart-like models that split data based on features; useful for explainability but prone to overfitting.
  • Random Forests: Are an ensemble of many Decision Trees, using 'bagging' and random feature selection to reduce overfitting and improve prediction accuracy.
  • Robustness: Random Forests are generally more robust and performant than single Decision Trees, making them a common choice for predictive tasks.
  • Interpretability vs. Performance: Opt for a simpler Decision Tree when explainability is crucial; choose Random Forests for higher predictive performance.
  • Versatility: Both algorithms are widely used for both classification and regression problems.

Code Example

python
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

# Load a sample dataset
iris = load_iris()
X, y = iris.data, iris.target

# Split data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Initialize and train a Random Forest Classifier
model = RandomForestClassifier(n_estimators=100, random_state=42)
model.fit(X_train, y_train)

# Make predictions and evaluate
predictions = model.predict(X_test)
accuracy = accuracy_score(y_test, predictions)
print(f"Random Forest Accuracy: {accuracy:.2f}")

How this code works

This code demonstrates how to build and evaluate a Random Forest model for classification, a common task in supervised learning. It begins by loading the load_iris dataset, a classic example featuring measurements of iris flowers and their corresponding species. The data is then systematically divided into distinct training and testing sets using train_test_split. This separation is crucial: the model learns from the training data and is subsequently assessed on the unseen test data to verify its ability to generalize to new examples. The test_size=0.3 parameter ensures 30% of the data is held back for this evaluation.

After preparing the data, a RandomForestClassifier is initialized. Here, n_estimators=100 tells the model to construct 100 individual decision trees. The model then learns patterns from the training features (X_train) and their labels (y_train) through the model.fit method. Once trained, model.predict generates predictions for the test set. Finally, accuracy_score measures how well these predictions match the actual labels (y_test), providing a clear performance metric. Notice the repeated use of random_state=42; this ensures that both the data split and the internal tree building process are consistent, making the results reproducible every time the code is run.

Related Articles

Hand-picked reads to go deeper on supervised learning algorithms.

Back to ML Engineer Roadmap