Phase 2: Classical Machine Learning

Handling Missing Values, Outliers & Imbalanced Data

Intermediate ~3 min read
Think of it this way A friendly analogy. Read this if the technical version feels dense. Show Hide

Imagine you're trying to bake your absolute favorite cake, but the recipe has some tricky spots. Maybe the page got wet, and one ingredient line is smudged, so you can't tell if it says "1 cup of sugar" or "2 cups." Or even worse, a whole ingredient is missing, like it just says "Add ____ flour." These are like "missing values" in computer talk. If you just guess wrong or skip it, your cake might turn out flat, too sweet, or totally inedible! It won't taste right, and your baking effort would be wasted. To fix this, you have a few options, just like a chef. If it's a small smudge, you might look at other similar recipes to see what they usually use for sugar and make your best educated guess. Most recipes might say "1 cup," so you go with that. If it's a huge, blank space, and you really can't figure out what's supposed to go there, sometimes you might have to sadly put that recipe aside and find a different one. These ways of figuring out what's missing or deciding not to use the messed-up parts help make sure your cake recipe is good and clear.

Now, what if your recipe did have all the ingredients, but one line was completely crazy? Like, it says "Add 100 cups of salt" to your sweet cake! That's clearly a huge mistake, right? That's what we call an "outlier" in the world of data. It's a piece of information that's so different from everything else it looks totally wrong, maybe because someone typed it in incorrectly, or there was a weird accident. If you actually put 100 cups of salt in your cake, it would be absolutely disgusting and ruin the whole thing! So, just like you wouldn't follow that "100 cups of salt" instruction, people working with data have special ways to spot these crazy numbers. They might see a measurement that's way too big or way too small compared to all the other measurements. Once they find these outliers, they have to decide if it was just a simple mistake that can be fixed (like changing "100" to "1 teaspoon"), or if it's so wild that it needs to be completely ignored so it doesn't mess up everything else.

All this careful checking and fixing of "missing values" and "outliers" might seem like extra work, but it's super important for making sure your cake (or, in computer terms, your amazing new program or smart AI) turns out perfectly. Just like a chef needs a good, clear recipe to bake a delicious cake every time, computers need really clean and accurate data to learn properly. So, when you see a self-driving car navigate safely, or a streaming service recommends a movie you love, a lot of careful work went into making sure the data they learned from was complete, correct, and didn't have any weird mistakes. This means you can trust the amazing things computers can do, because the information they use is top-notch!

Addressing missing values, outliers, and imbalanced data is crucial in Feature Engineering to ensure robust and accurate machine learning models. Missing values, common due to data entry errors, non-responses, or corruption, can break models or introduce bias. Practical strategies range from simple deletion (if missingness is minimal and random) to imputation. For numerical data, median imputation is often preferred over mean as it's less sensitive to outliers. For categorical data, mode imputation is typical. More advanced methods like K-NN or regression imputation offer greater accuracy but come with higher computational cost and complexity. The choice depends heavily on the extent and pattern of missingness, as well as the nature of your data.

Outliers are data points significantly different from others, potentially caused by measurement errors, genuine extreme events, or data entry mistakes. They can disproportionately influence model training, skew statistics, and lead to poor generalization, especially for models sensitive to scale like linear regression. Detection often involves statistical methods like the Z-score or Interquartile Range (IQR), or visual inspection using box plots and scatter plots. Treatment options include capping (Winsorization) by replacing extreme values with a specified percentile, data transformations (e.g., log transformation to reduce spread), or, in rare cases, outright deletion if they are confirmed errors. It's vital to understand if an outlier is a true anomaly or a critical data point before treatment, as removing valuable information can be detrimental.

Imbalanced data occurs when one class significantly outnumbers another, a common scenario in fraud detection or rare disease diagnosis. Models trained on imbalanced datasets tend to become biased towards the majority class, performing poorly on the minority class—which is often the class of primary interest. Strategies to counter this include resampling techniques like oversampling the minority class (e.g., SMOTE, which synthesizes new examples) or undersampling the majority class. Algorithmic approaches involve using cost-sensitive learning, where misclassifying the minority class incurs a higher penalty, or ensemble methods designed for imbalance. Furthermore, evaluating model performance with appropriate metrics like F1-score, precision, recall, or ROC-AUC is crucial, as accuracy alone can be misleading in imbalanced scenarios.

Key Takeaways

  • Missing values: Always address them, choosing imputation (median/mode for simplicity, advanced for accuracy) or deletion based on data context and amount.
  • Outliers: Detect using statistical methods/visualizations, then decide on capping, transformation, or deletion based on their cause and impact on the model.
  • Imbalanced data: Use resampling (oversampling/undersampling) or algorithmic techniques to ensure models learn from the critical minority class.
  • Context is paramount: The best strategy for each issue is highly dependent on your specific dataset, domain knowledge, and the goals of your ML problem.
  • Metrics matter: For imbalanced data, rely on F1-score, precision, recall, or ROC-AUC, not just accuracy, to properly assess model performance.

Code Example

python
import pandas as pd
import numpy as np
from sklearn.impute import SimpleImputer

# Sample DataFrame with missing values and potential outliers
data = {'Age': [25, 30, np.nan, 40, 150, 35],
        'Salary': [50000, np.nan, 75000, 80000, 60000, 55000]}
df = pd.DataFrame(data)

print("Original DataFrame:\n", df)

# 1. Handle Missing Values (Median Imputation for numerical features)
imputer = SimpleImputer(strategy='median')
df[['Age', 'Salary']] = imputer.fit_transform(df[['Age', 'Salary']])

# 2. Simple Outlier Capping (e.g., cap Age at 90)
df['Age'] = np.where(df['Age'] > 90, 90, df['Age'])

print("\nDataFrame after imputation and outlier capping:\n", df)

How this code works

This code demonstrates fundamental feature engineering steps: handling missing data and mitigating the impact of outliers. It begins by setting up a sample dataset df that intentionally includes np.nan (missing values) and an extreme 'Age' value (150) to simulate common imperfections found in real-world data.

The process first addresses missing values using SimpleImputer(strategy='median'). This imputes, or fills in, the np.nan entries in 'Age' and 'Salary' with the median value calculated from the respective non-missing data in each column. Choosing 'median' over other strategies like 'mean' is a subtle but important decision here because the median is less sensitive to extreme values, making it a more robust choice when outliers might be present (even before they are explicitly handled). Next, outlier capping is applied to the 'Age' column. The np.where function checks if any 'Age' is above 90; if so, it replaces that value with 90, effectively limiting extreme high ages without removing the entire data point. This sequence ensures the data is clean and more reliable for subsequent analysis.

Related Articles

Hand-picked reads to go deeper on feature engineering.

Back to ML Engineer Roadmap