Linear Regression and Logistic Regression are foundational algorithms in supervised learning, serving distinct yet complementary purposes. Linear Regression is your go-to for predicting continuous numerical values. Think predicting house prices based on size and location, or estimating a person's age based on certain features. It models a linear relationship between input features and the target variable. In contrast, Logistic Regression, despite its name, is primarily a classification algorithm. Its strength lies in predicting categorical outcomes, most commonly binary ones, like whether an email is spam or not, if a customer will churn, or if a transaction is fraudulent. While both aim to understand feature relationships, their output types dictate when to use which.
Under the hood, both algorithms attempt to learn a set of weights (coefficients) and a bias term that best describes the relationship between your input features and the target. Linear Regression directly fits a straight line (or a hyperplane in higher dimensions) through the data points, minimizing the sum of squared differences between the predicted and actual values. This line represents the learned relationship. Logistic Regression also starts with a linear combination of inputs and weights, but then it passes this result through a special non-linear function called the sigmoid function. The sigmoid squashes any real-valued number into a probability score between 0 and 1, which can then be thresholded (e.g., above 0.5 is one class, below 0.5 is the other) to make a final classification decision.
From a practical ML engineer's perspective, both linear and logistic regression are invaluable for their simplicity, interpretability, and efficiency. They serve as excellent baselines for more complex models and are often the first algorithms you'll try. Their coefficients can directly tell you the impact of each feature on the outcome, making them highly interpretable for stakeholders. While they assume a linear relationship and might struggle with highly complex, non-linear data, they are robust, computationally inexpensive, and effective for a wide array of real-world problems. Mastering these provides a strong bedrock for understanding more advanced supervised learning techniques.
Key Takeaways
- Linear Regression predicts continuous numerical values (regression task).
- Logistic Regression predicts categorical (often binary) outcomes (classification task).
- Both learn feature weights and bias, modeling linear relationships.
- Logistic Regression uses a sigmoid function to output probabilities for classification.
- They are fundamental, interpretable, and serve as strong baseline models.
Code Example
import numpy as np
from sklearn.linear_model import LinearRegression, LogisticRegression
# Sample Data (simplified for brevity)
X = np.array([[1, 2], [2, 3], [3, 4], [4, 5], [5, 6]])
y_linear = np.array([3, 5, 7, 9, 11]) # e.g., y = 2x - 1
y_logistic = np.array([0, 0, 1, 1, 1]) # Binary classification
# Linear Regression Example
lin_model = LinearRegression()
lin_model.fit(X, y_linear)
print(f"Linear Prediction for [6,7]: {lin_model.predict(np.array([[6,7]]))}")
# Logistic Regression Example
log_model = LogisticRegression(solver='liblinear')
log_model.fit(X, y_logistic)
print(f"Logistic Prediction for [2,3]: {log_model.predict(np.array([[2,3]]))}")
print(f"Logistic Prob for [2,3]: {log_model.predict_proba(np.array([[2,3]]))}")How this code works
This code demonstrates how to apply LinearRegression and LogisticRegression, two core supervised learning algorithms, for different prediction tasks. It begins by importing numpy for numerical data handling and the specific regression models from sklearn.linear_model. Sample data is then prepared: X represents features, y_linear provides continuous targets suitable for linear regression, and y_logistic offers binary targets (0s and 1s) for classification.
For LinearRegression, an lin_model is created and trained using lin_model.fit(X, y_linear). This process teaches the model to find the optimal straight line (or plane) that best fits the input features to their continuous outputs. Once trained, lin_model.predict makes a new prediction. Similarly, log_model = LogisticRegression(solver='liblinear') sets up the Logistic Regression model. The solver='liblinear' parameter is important here; it specifies an efficient algorithm particularly well-suited for smaller datasets and binary classification, helping the model learn effectively. After log_model.fit(X, y_logistic) trains the classification model, log_model.predict provides the direct class output, while log_model.predict_proba reveals the probability of an input belonging to each class.