Phase 2: Classical Machine Learning

Interaction & Polynomial Features

Intermediate ~3 min read
Think of it this way A friendly analogy. Read this if the technical version feels dense. Show Hide

You know how when you're baking a cake, sometimes a basic recipe is good, but it doesn't quite make it perfect? Maybe it's a little too dry, or not quite sweet enough. In programming, we sometimes have "recipes" that try to predict things, like a new game's popularity based on its cost or ad campaigns. But just like with cakes, these basic recipes sometimes aren't smart enough to capture all the yummy details.

That's where making special "new ingredients" comes in. Imagine we have an ingredient called "Sweetness" (like how much sugar we add). A simple recipe might just say, "add more sweetness for a tastier cake." But what if, after a certain point, more sweetness actually makes the cake less delicious, or even a little sickly? To handle this, we can create a "Super Sweetness" ingredient. This isn't just adding more sugar; it's a special instruction that helps our recipe understand that the impact of sweetness changes. It helps our recipe learn that while a cake gets tastier as it gets sweeter, it might also hit a peak, and even start to taste worse if it gets too sweet. This allows the recipe to find the ideal amount.

Then there are "mixed ingredients," which are even cooler. Think about making a special chocolate cake. You have "Chocolate Amount" and "Spice Level" (maybe a tiny bit of chili powder, like in some fancy chocolate bars!). A basic recipe might just say, "add chocolate, add spice." But what if the magic really happens when the chocolate and spice are combined? Maybe a little spice makes the chocolate flavor pop in an amazing way that neither could do alone. We can create a new "Chocolate-Chili Fusion" ingredient by multiplying the "Chocolate Amount" and "Spice Level" together. This new "Chocolate-Chili Fusion" ingredient helps our recipe understand that some ingredients work together to create a unique taste that's more than just the sum of their parts.

So, by creating these clever "Super Sweetness" and "Chocolate-Chili Fusion" ingredients from our basic ones, we can make our prediction "recipes" much smarter. This means when you build something that tries to understand how different things connect, like predicting the best temperature for a plant to grow or the perfect amount of playtime for a pet to be happy, you can teach it to see these deeper, hidden patterns, making its predictions much more accurate and helpful.

Polynomial features are a technique to create new features by raising existing features to a specific power. For example, if you have a feature 'Age', you can create 'Age^2' (Age squared) or 'Age^3' (Age cubed). The primary goal here is to enable linear models to capture non-linear relationships within your data. Imagine a scenario where a target variable (like salary) initially increases with age but then starts to level off or even decrease past a certain point. A simple linear relationship (Age * coefficient) wouldn't capture this curve, but adding 'Age^2' allows the model to learn a parabolic relationship, making it more flexible and potentially more accurate.

Interaction features, on the other hand, are created by multiplying two or more existing features together. For instance, if you have 'Advertising Spend' and 'Discount Percentage', an interaction feature could be 'Advertising Spend * Discount Percentage'. This type of feature is crucial when the effect of one feature on the target variable is not independent but rather contingent on the value of another feature. Perhaps high advertising spend only translates to significantly higher sales when a product is also heavily discounted. Without an interaction term, a model might struggle to understand this synergistic or conditional relationship, treating the effects of 'Advertising Spend' and 'Discount Percentage' as purely additive.

Both polynomial and interaction features are powerful tools in a feature engineer's arsenal, especially when working with models that inherently assume linearity, such as linear regression, logistic regression, or support vector machines with a linear kernel. By explicitly crafting these higher-order and interactive terms, you're essentially providing the model with new "views" of the data that might reveal underlying patterns it couldn't discover on its own. However, exercise caution: generating many polynomial and interaction features rapidly increases your dataset's dimensionality, which can lead to increased computational cost, introduce multicollinearity, and heighten the risk of overfitting, especially with high degrees. Always consider domain knowledge and validate your features carefully.

Key Takeaways

  • Polynomial features create non-linear relationships (e.g., curves) from linear features.
  • Interaction features capture conditional effects where one feature's impact depends on another.
  • They enhance linear models' ability to fit complex data patterns.
  • Mind the trade-offs: increased dimensionality, computational cost, and risk of overfitting.

Code Example

python
import numpy as np
from sklearn.preprocessing import PolynomialFeatures

# Original features: e.g., [feature1, feature2]
X = np.array([["2, 3"],
              ["4, 5"]])

# Create a PolynomialFeatures transformer
# degree=2 includes original features, their squares, and their interaction
# include_bias=False means it won't add a column of ones (intercept)
poly = PolynomialFeatures(degree=2, include_bias=False)

# Transform the features
X_poly = poly.fit_transform(X)

print("Original Features:")
print(X)
print("\nTransformed Features (Polynomial and Interaction):")
print(X_poly)

# Example output for X = [[2,3]]: [2, 3, 2^2, 3^2, 2*3] -> [2, 3, 4, 9, 6]

How this code works

This code demonstrates how to enrich a dataset by creating new features from existing ones, a process crucial for helping machine learning models uncover complex, non-linear relationships in data. It uses sklearn.preprocessing.PolynomialFeatures to automatically generate these new, higher-order features and interaction terms.

Initially, X holds the original input data. A PolynomialFeatures object is then configured with degree=2, which means it will generate original features, their squares (e.g., feature1^2), and all two-way interaction terms (e.g., feature1 * feature2). The include_bias=False setting is important because it prevents adding an extra column of ones (an intercept term), as this is typically handled by the machine learning model itself. Finally, poly.fit_transform(X) applies these defined transformations, creating X_poly. A subtle but critical point for beginners is that PolynomialFeatures expects purely numeric input arrays. The X in this example, structured as an array of strings like "2, 3", would actually cause fit_transform to error or produce incorrect results if not first converted into proper numeric types, unlike the [2, 3, 4, 9, 6] example output which assumes numeric inputs. This preprocessing step is often a prerequisite for using PolynomialFeatures effectively.

Related Articles

Hand-picked reads to go deeper on feature engineering.

Back to ML Engineer Roadmap