Overfitting & Underfitting
Description
Overfitting and underfitting are common problems encountered during training deep learning models that affect their ability to generalize well to unseen data.
- Overfitting: Occurs when a model learns the training data too well, including noise and outliers. It performs excellently on training data but poorly on new, unseen data.
- Underfitting: Happens when a model is too simple to capture the underlying patterns of the data, resulting in poor performance on both training and test datasets.
Balancing model complexity and training duration, along with proper regularization, helps avoid both overfitting and underfitting.
Examples
Python example showing underfitting and overfitting with polynomial regression:
import numpy as np
import matplotlib.pyplot as plt
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline
# Generate synthetic data
np.random.seed(0)
X = np.sort(np.random.rand(30) * 10).reshape(-1, 1)
y = np.sin(X).ravel() + np.random.normal(0, 0.3, X.shape[0])
# Underfitting model (degree 1)
model_underfit = make_pipeline(PolynomialFeatures(degree=1), LinearRegression())
model_underfit.fit(X, y)
# Overfitting model (degree 15)
model_overfit = make_pipeline(PolynomialFeatures(degree=15), LinearRegression())
model_overfit.fit(X, y)
# Plot results
X_plot = np.linspace(0, 10, 100).reshape(-1, 1)
plt.scatter(X, y, color='black', label='Data')
plt.plot(X_plot, model_underfit.predict(X_plot), label='Underfitting (Degree 1)', color='blue')
plt.plot(X_plot, model_overfit.predict(X_plot), label='Overfitting (Degree 15)', color='red')
plt.legend()
plt.title("Underfitting vs Overfitting Example")
plt.show()Real-World Applications
Image Recognition
Properly tuned models avoid overfitting on training images to generalize well on unseen images.
Natural Language Processing
Balancing model complexity prevents underfitting simple language tasks or overfitting to training text corpora.
Medical Diagnosis
Models must generalize well on new patient data, avoiding overfitting to specific cases in training.
Financial Forecasting
Avoiding overfitting ensures predictions stay reliable on volatile market data.
Resources
Recommended Books
- Deep Learning by Ian Goodfellow
- Hands-On Machine Learning by Aurélien Géron
- CS231n: Overfitting & Regularization
Interview Questions
What is overfitting and how can it be prevented?
Overfitting occurs when a model learns noise or details from training data that don't generalize to new data. It can be prevented by techniques like regularization, dropout, early stopping, and using more training data.
How do you detect underfitting in a model?
Underfitting is detected when the model performs poorly on both training and validation data, indicating it is too simple to capture the data patterns.
What methods help in balancing between underfitting and overfitting?
Using techniques like cross-validation, adjusting model complexity, regularization, early stopping, and data augmentation help balance underfitting and overfitting.