Phase 2: Classical Machine Learning

Precision, Recall, F1, AUC-ROC & Log Loss

Intermediate ~2 min read
Think of it this way A friendly analogy. Read this if the technical version feels dense. Show Hide

Imagine you have a super smart robot whose job is to help you sort cookies. Out of a huge batch, your robot needs to find all the "perfectly baked" ones – not too burnt, not too raw, just right. Sometimes, if you only ask, "How many cookies did you get right overall?", it might seem like your robot is doing a great job. But what if it mostly just correctly sorted the really burnt cookies, and missed all the perfect ones? We need better ways to understand if our robot is truly excellent at finding exactly what we want.

That's where something called "Precision" comes in. Think of Precision as how trustworthy your robot is when it makes a positive claim. When your robot says a cookie is perfectly baked and puts it in the "perfect" pile, how often is it actually perfect? If your robot has high Precision, it means that almost every cookie it labels as "perfect" truly is perfect. This is super important if the cost of a mistake is high, like if you're serving these cookies to guests and you absolutely don't want to give them a burnt one you claimed was perfect!

Then there's "Recall," which is about how thorough your robot is. This asks: Of all the cookies that are truly perfectly baked in the whole batch, how many did your robot actually find and put in the "perfect" pile? A robot with high Recall is great at finding nearly all the perfect cookies, so none get missed. This is important if missing something good is a big deal, like accidentally throwing away a delicious, perfectly baked cookie because your robot didn't spot it.

Sometimes, a robot might be great at Precision but not Recall, or vice-versa. For example, it might only pick cookies it's absolutely sure are perfect (high Precision), but miss many others. Or it might pick almost every perfect cookie (high Recall), but also include a lot of burnt ones by mistake (low Precision). That's where the "F1-Score" (we just call it "F1") helps. F1 gives you one balanced number that tells you how good your robot is at both being trustworthy and thorough at the same time. So, when you build your own smart helper programs someday, you won't just ask "is it right?" You'll ask, "is it trustworthy and thorough?" This helps you decide if your helper is doing a truly great job, especially for important tasks like finding good cookies, or even more serious things like finding things that truly matter.

When evaluating classification models, relying solely on accuracy can be misleading, especially with imbalanced datasets where one class is far more common. This is where Precision, Recall, and F1-Score become indispensable. Precision answers: "Of all the instances our model predicted as positive, how many were actually positive?" It's crucial when the cost of a false positive is high, for example, incorrectly flagging a healthy patient with a disease. Recall, conversely, asks: "Of all the actual positive instances, how many did our model correctly identify?" This is vital when the cost of a false negative is high, such as missing a fraudulent transaction or a cancerous tumor. Often, there's a trade-off between these two. The F1-Score provides a single metric that balances both, being the harmonic mean of Precision and Recall, making it a robust choice when you need a model that performs well on both fronts.

Key Takeaways

  • Precision minimizes false positives (e.g., incorrect spam flag).
  • Recall minimizes false negatives (e.g., missed fraud detection).
  • F1-Score provides a balance, useful for imbalanced datasets.
  • AUC-ROC evaluates overall model discrimination across all thresholds.
  • Log Loss measures the confidence and accuracy of probability predictions.

Code Example

python
from sklearn.metrics import precision_score, recall_score, f1_score, roc_auc_score, log_loss
import numpy as np

# Example data: True labels and predicted probabilities
y_true = np.array([0, 1, 1, 0, 1, 0, 1, 1, 0, 0])
y_pred_proba = np.array([0.1, 0.8, 0.7, 0.2, 0.9, 0.3, 0.6, 0.75, 0.4, 0.25])

# For Precision, Recall, F1, we need binary predictions (e.g., threshold at 0.5)
y_pred_binary = (y_pred_proba >= 0.5).astype(int)

print(f"Precision: {precision_score(y_true, y_pred_binary):.2f}")
print(f"Recall: {recall_score(y_true, y_pred_binary):.2f}")
print(f"F1 Score: {f1_score(y_true, y_pred_binary):.2f}")
print(f"AUC-ROC: {roc_auc_score(y_true, y_pred_proba):.2f}")
print(f"Log Loss: {log_loss(y_true, y_pred_proba):.2f}")

How this code works

This code demonstrates how to calculate essential machine learning model evaluation metrics using sklearn.metrics. It starts by setting up y_true, the actual labels (0s or 1s), and y_pred_proba, the model's predicted probabilities for each outcome. Crucially, for precision_score, recall_score, and f1_score, the probabilistic predictions need to be converted into discrete binary classifications. This is achieved by creating y_pred_binary, where any probability 0.5 or greater becomes a 1, and anything below becomes a 0. This fixed threshold of 0.5 is a common choice, but adjusting it would directly impact these scores. The code then prints these classification-based metrics, which rely on the converted binary predictions.

In contrast, roc_auc_score and log_loss directly leverage the y_pred_proba values. roc_auc_score evaluates how well the model ranks positive instances above negative ones across all possible classification thresholds, providing a comprehensive view of its discriminative power. log_loss quantifies the accuracy of the predicted probabilities themselves, penalizing incorrect confident predictions more heavily. These probabilistic metrics offer a deeper insight into model performance beyond simple true/false classifications, and the :.2f formatting ensures clear, rounded output.

Related Articles

Hand-picked reads to go deeper on model evaluation & validation.

Back to ML Engineer Roadmap