Phase 2: Classical Machine Learning

Learning Curves & Calibration Plots

Intermediate ~3 min read
Think of it this way A friendly analogy. Read this if the technical version feels dense. Show Hide

You know how when you're learning something new, like cooking a tricky recipe, you don't get it perfect right away? It takes practice! Sometimes, even after lots of practice, you make the same mistakes, or you're good for yourself but struggle when cooking for a party. In coding, when we teach a computer model to do a job, like recognizing pictures of cats, we need to see how well it's really learning and improving. That's where "learning curves" come in – they're like a progress report to understand our model's journey.

Imagine learning to bake cookies. A "learning curve" is a chart showing two things: how good your cookies are when practicing by yourself (the computer's "training error"), and how good they are for friends or a bake sale (the "validation error" – seeing if your skills work on new things). We plot these against how many times you've practiced. If your cookies are always burnt or too salty, even after many practices, and also bad for friends, your basic recipe might be too simple. This is like a computer model that "underfits" – it hasn't learned enough, no matter how much data it sees. The fix isn't more practice, but a better, more detailed recipe!

On the flip side, what if you've practiced one specific cookie recipe so many times that you can make it perfectly for yourself every time? Your "practice cookies" are amazing. But then, when you make them for a big family gathering, maybe with slightly different ingredients or a different oven, they suddenly turn out wonky or not quite right. This is like a computer model that "overfits." It got too good at remembering the exact details of its practice data and struggles to adapt to new situations. Here, your "practice cookie" score would be super low (great!), but your "friends' cookie" score much higher (not so great!). To fix this, you might need to practice with more varied ingredients, or learn general baking principles instead of just one exact recipe.

So, when you're building a computer program to learn something, looking at these "learning curves" helps you become a better "chef" of your code. By seeing how your model's "practice errors" and "real-world errors" change with more experience, you can quickly figure out if your model needs a more advanced "recipe" or if it needs to learn to be more flexible and not just memorize. This means you can make smart decisions to help your model perform its best, whether it's identifying cats or predicting the weather!

When evaluating a machine learning model, simply looking at a single metric like accuracy or F1-score doesn't always tell the full story. Learning curves are powerful diagnostic tools that help you understand your model's performance by plotting its training error and validation error against the amount of training data used. By visualizing how these errors change as your model sees more data, you can diagnose fundamental issues like high bias (underfitting) or high variance (overfitting), guiding your next steps in model improvement.

Interpreting learning curves is key. If both the training and validation error are high and converge at a relatively high error rate, your model likely suffers from high bias (underfitting). This means the model is too simple for the data, and even adding more data won't help much; you'd need a more complex model or better features. Conversely, if the training error is very low but the validation error is significantly higher, with a large gap between the curves, your model is experiencing high variance (overfitting). In this scenario, the model has memorized the training data too well; adding more data, applying regularization, or simplifying the model could be beneficial.

Beyond just classifying instances correctly, sometimes you need to trust your model's predicted probabilities. This is where calibration plots, also known as reliability diagrams, come in. A calibration plot assesses how well your model's predicted probabilities match the actual proportion of positive outcomes for different probability bins. For instance, if your model predicts a 70% chance of an event happening for 100 samples, a perfectly calibrated model would see that event truly happen in approximately 70 of those samples. A perfectly calibrated model's predictions would lie along the diagonal line on the plot. Models like Logistic Regression tend to be well-calibrated by default, while others, such as Naive Bayes or uncalibrated Support Vector Machines, often require post-hoc calibration techniques (e.g., Isotonic Regression, Platt Scaling) to make their probability outputs more trustworthy for critical decision-making where the "likelihood" itself is as important as the final prediction.

Key Takeaways

  • Learning curves diagnose bias (underfitting) and variance (overfitting) by plotting error against the amount of training data.
  • High bias manifests as both training and validation errors converging at a high level; high variance shows a large gap between low training error and high validation error.
  • Calibration plots assess the reliability of a model's predicted probabilities, comparing them to actual observed frequencies.
  • A perfectly calibrated model's probability predictions align with the true proportion of positive outcomes, visualized as a diagonal line on the plot.
  • Use learning curves to inform decisions on model complexity or data quantity; use calibration plots when the trustworthiness of probability estimates is crucial (e.g., risk assessment).

Code Example

python
import numpy as np
import matplotlib.pyplot as plt
from sklearn.naive_bayes import GaussianNB
from sklearn.model_selection import learning_curve
from sklearn.datasets import load_digits

# Load a sample dataset
X, y = load_digits(return_X_y=True)

# Create a classifier
clf = GaussianNB()

# Generate learning curves
train_sizes, train_scores, test_scores = learning_curve(
    clf, X, y, cv=5, n_jobs=-1,
    train_sizes=np.linspace(0.1, 1.0, 10))

train_scores_mean = np.mean(train_scores, axis=1)
test_scores_mean = np.mean(test_scores, axis=1)

plt.plot(train_sizes, train_scores_mean, 'o-', label="Training score")
plt.plot(train_sizes, test_scores_mean, 'o-', label="Cross-validation score")
plt.title("Learning Curve")
plt.xlabel("Training examples")
plt.ylabel("Score")
plt.legend(loc="best")
plt.grid()
plt.show()

How this code works

This code generates a learning curve plot, a vital tool for understanding how a model's performance changes with varying amounts of training data. It helps diagnose if a model is underfitting (potentially needs more complexity or features) or overfitting (might benefit from more data or regularization).

First, essential libraries like numpy for calculations, matplotlib.pyplot for plotting, and sklearn components are imported. A sample dataset, load_digits, is loaded, and a GaussianNB classifier is initialized. The core logic resides in sklearn.model_selection.learning_curve. This function repeatedly trains and evaluates the clf model using different subsets of the training data, specified by train_sizes (ranging from 10% to 100% of the data in 10 steps using np.linspace). Crucially, for each training subset size, learning_curve performs 5-fold cross-validation (cv=5). This means it internally splits that subset five times into training and validation folds, producing multiple train_scores and test_scores. These scores are then averaged using np.mean(..., axis=1) to get a single performance value for each training size.

Finally, matplotlib.pyplot is used to plot these averaged training and cross-validation scores against the train_sizes. The plot includes labels, a title, a legend, and a grid for clarity before plt.show() displays it. A subtle but important detail is how learning_curve handles the evaluation: it doesn't just apply cv to the full dataset. Instead, it creates smaller subsets of the training data first, and then applies cross-validation within each of those subsets. This comprehensive internal process ensures a robust assessment of performance across different data volumes, which is key to understanding the model's behavior.

Related Articles

Hand-picked reads to go deeper on model evaluation & validation.

Back to ML Engineer Roadmap