Home / Topics / Underfitting vs Overfitting

Underfitting vs Overfitting

Description

Underfitting and overfitting are common problems in machine learning that occur when a model fails to generalize well from training data to unseen data.

Underfitting

Underfitting happens when a model is too simple to capture the underlying structure of the data. It performs poorly on both training and test data, indicating that it hasn't learned enough from the data.

  • Model has high bias and low variance
  • Fails to capture important patterns
  • Leads to poor training and testing performance
  • Example: Using a linear model for highly nonlinear data

Overfitting

Overfitting occurs when a model learns not only the underlying pattern but also the noise in the training data. This results in excellent training performance but poor generalization to new, unseen data.

  • Model has low bias and high variance
  • Captures noise as if it were a meaningful pattern
  • Leads to excellent training but poor testing performance
  • Example: Deep decision trees or high-degree polynomial regression on small datasets

Comparison Summary

Aspect Underfitting Overfitting
Model Complexity Too simple Too complex
Training Error High Low
Test Error High High
Bias High Low
Variance Low High

Examples

Python Example: Visualizing Underfitting and Overfitting with Polynomial Regression

import numpy as np
import matplotlib.pyplot as plt
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error

# Generate data
np.random.seed(42)
X = np.sort(np.random.rand(30, 1) * 10, axis=0)
y = np.sin(X).ravel() + np.random.normal(0, 0.3, X.shape[0])

degrees = [1, 4, 15]  # 1: underfitting, 4: good fit, 15: overfitting
plt.figure(figsize=(18, 5))

for i, degree in enumerate(degrees, 1):
    poly = PolynomialFeatures(degree)
    X_poly = poly.fit_transform(X)
    
    model = LinearRegression()
    model.fit(X_poly, y)
    y_pred = model.predict(X_poly)
    
    mse = mean_squared_error(y, y_pred)
    plt.subplot(1, 3, i)
    plt.scatter(X, y, color='blue', label='Data')
    plt.plot(X, y_pred, color='red', label=f'Degree {degree}')
    plt.title(f'Polynomial Degree {degree}\nMSE: {mse:.2f}')
    plt.legend()

plt.show()

Real-World Applications

Handling Underfitting and Overfitting

  • Underfitting Solutions: Use more complex models, add relevant features, reduce regularization.
  • Overfitting Solutions: Use regularization techniques (L1, L2), prune decision trees, gather more training data, use dropout in neural networks.
  • Model Validation: Employ cross-validation techniques to detect and avoid both problems.
  • Early Stopping: Stop training when validation error starts to increase to prevent overfitting.
Underfitting vs Overfitting Concept

Resources

The following resources will be manually added later:

Video Tutorials

YouTube video link.

PDF/DOC Materials

Drive links for PDF/DOC files .

Interview Questions

1. What is the difference between underfitting and overfitting?

Show Answer

Underfitting happens when a model is too simple and fails to capture the data's patterns, resulting in poor performance on both training and test data. Overfitting happens when a model is too complex and captures noise as if it were a pattern, resulting in good training but poor test performance.

2. How can you detect overfitting?

Show Answer

Overfitting can be detected when the model shows very low error on training data but significantly higher error on validation or test data.

3. What techniques can be used to prevent overfitting?

Show Answer

Techniques include using regularization (L1, L2), early stopping, pruning trees, dropout in neural networks, and increasing training data.

4. Why is underfitting problematic?

Show Answer

Underfitting is problematic because the model cannot capture the important patterns in data, resulting in poor predictions and performance even on training data.

Back to Topics