Polynomial Regression in Machine Learning

1. Introduction to Polynomial Regression

Polynomial Regression is an extension of Linear Regression that models the relationship between the independent variable (X) and the dependent variable (Y) as an n-degree polynomial function. It is used when data exhibits a non-linear relationship, meaning that a straight line (Linear Regression) is not sufficient to capture the trend.

💡 Example Use Cases:
✔ Predicting growth patterns in economics.
✔ Modeling non-linear trends in real estate pricing.
✔ Forecasting weather patterns or climate change models.
✔ Fitting complex biological or chemical data relationships.

2. Why Do We Need Polynomial Regression?

Linear Regression assumes that the relationship between X and Y is strictly linear, but real-world data often follows a curved pattern.

✅ Polynomial Regression captures this curvature by adding higher-order terms (X², X³, …) to the regression equation.

🔹 Example: Suppose you want to predict house prices based on square footage, but the relationship between square footage and price is not perfectly linear. Polynomial Regression can model this complex trend better than a simple straight line.

3. Mathematical Representation of Polynomial Regression

The equation for Polynomial Regression extends Linear Regression by including higher-degree polynomial terms: Y=b0+b1X+b2X2+b3X3+…+bnXnY = b_0 + b_1X + b_2X^2 + b_3X^3 + … + b_nX^n

Where:

Y = Dependent variable (predicted value)
X = Independent variable (input feature)
b₀, b₁, b₂, …, bₙ = Regression coefficients
X², X³, …, Xⁿ = Higher-order polynomial terms

📌 Example:

A quadratic polynomial regression (degree 2) equation: Y=b0+b1X+b2X2Y = b_0 + b_1X + b_2X^2
A cubic polynomial regression (degree 3) equation: Y=b0+b1X+b2X2+b3X3Y = b_0 + b_1X + b_2X^2 + b_3X^3

4. Choosing the Degree of the Polynomial

The degree (n) of the polynomial determines how flexible the curve is:

Degree = 1 (Linear Regression) → Straight line.
Degree = 2 (Quadratic Regression) → Parabolic curve.
Degree = 3 (Cubic Regression) → More complex curvature.
Degree = n (Higher-Order Regression) → Captures even finer details but risks overfitting.

✅ Higher degrees can fit the data better, but they also increase complexity and risk overfitting.

5. Underfitting vs. Overfitting in Polynomial Regression

🔹 Underfitting (High Bias, Low Variance)

When the polynomial degree is too low (e.g., degree = 1), the model fails to capture the trend in the data.

🔹 Overfitting (Low Bias, High Variance)

When the polynomial degree is too high (e.g., degree = 10), the model fits the training data too well but fails on new test data.

✅ Choosing the right degree is key! Use techniques like Cross-Validation to find the optimal balance.

6. Cost Function in Polynomial Regression

Polynomial Regression uses the Mean Squared Error (MSE) to measure the difference between predicted and actual values: MSE=1n∑(Yi−Yi^)2MSE = \frac{1}{n} \sum (Y_i – \hat{Y_i})^2

Where:

Yᵢ = Actual values
Ŷᵢ = Predicted values

🎯 Goal: Minimize the MSE to improve model accuracy.

7. Implementing Polynomial Regression in Python (Sklearn)

Let’s implement Polynomial Regression using Python and Scikit-Learn.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error, r2_score

# Generate synthetic dataset
np.random.seed(42)
X = 6 * np.random.rand(100, 1) - 3  # Random values between -3 and 3
y = 2 + X - 2*X**2 + 0.5*X**3 + np.random.randn(100, 1)  # Cubic relationship with noise

# Split data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Transform input features to polynomial features (degree = 3)
poly_features = PolynomialFeatures(degree=3, include_bias=False)
X_train_poly = poly_features.fit_transform(X_train)
X_test_poly = poly_features.transform(X_test)

# Train Polynomial Regression model
model = LinearRegression()
model.fit(X_train_poly, y_train)

# Predict using the trained model
y_pred = model.predict(X_test_poly)

# Evaluate model
mse = mean_squared_error(y_test, y_pred)
r2 = r2_score(y_test, y_pred)

print(f'Mean Squared Error: {mse:.2f}')
print(f'R² Score: {r2:.2f}')

# Visualize Polynomial Regression Fit
X_curve = np.linspace(-3, 3, 100).reshape(-1, 1)
X_curve_poly = poly_features.transform(X_curve)
y_curve = model.predict(X_curve_poly)

plt.scatter(X_test, y_test, color="blue", label="Actual Data")
plt.plot(X_curve, y_curve, color="red", linewidth=2, label="Polynomial Fit")
plt.xlabel("X")
plt.ylabel("Y")
plt.legend()
plt.show()

8. Model Evaluation Metrics for Polynomial Regression

✅ 1. R-squared Score (R²)

Measures how well the model explains variance in the dependent variable.
R² closer to 1 → Better model fit.

✅ 2. Mean Absolute Error (MAE)

MAE=1n∑∣Yi−Yi^∣MAE = \frac{1}{n} \sum |Y_i – \hat{Y_i}|

Measures absolute differences between actual and predicted values.

✅ 3. Mean Squared Error (MSE)

MSE=1n∑(Yi−Yi^)2MSE = \frac{1}{n} \sum (Y_i – \hat{Y_i})^2

Penalizes large errors more than MAE.

✅ 4. Root Mean Squared Error (RMSE)

RMSE=MSERMSE = \sqrt{MSE}

Provides an error measure in the same unit as Y.

9. Advantages and Disadvantages of Polynomial Regression

✅ Advantages:

✔ Captures non-linear relationships effectively.
✔ More flexible than Linear Regression.
✔ Still interpretable when using lower-degree polynomials.

❌ Disadvantages:

❌ Prone to overfitting if the polynomial degree is too high.
❌ Extrapolation is unreliable (unpredictable outside training range).
❌ Requires careful feature engineering and degree selection.

10. When to Use Polynomial Regression?

✅ When data shows a clear curved trend that cannot be captured by Linear Regression.
✅ When interpretability is still important (not as complex as deep learning models).
✅ When you need a balance between flexibility and simplicity.

11. Summary

✔ Polynomial Regression extends Linear Regression by including polynomial terms.
✔ Higher-degree polynomials capture more complex relationships but risk overfitting.
✔ Evaluated using MSE, R² Score, RMSE, and MAE.
✔ Python implementation using Scikit-Learn (PolynomialFeatures, LinearRegression).

Mastering Polynomial Regression is a key step in understanding non-linear patterns in machine learning!