š Introduction
Overfitting and Underfitting are two common problems encountered while training Machine Learning (ML) models. A successful Machine Learning model should not only perform well on the training data but should also generalize effectively to new, unseen data. Overfitting occurs when a model learns the training data too closely, while underfitting occurs when a model fails to learn important patterns from the data.
Information
š Overview
šÆ What Is Underfitting?
Underfitting occurs when a Machine Learning model is too simple to capture the underlying patterns in the data. As a result, it performs poorly on both the training dataset and unseen test data.
Characteristics
- High training error.
- High testing error.
- Fails to learn important relationships.
- Often caused by an overly simple model.
Tip
š Causes of Underfitting
- Using an overly simple algorithm.
- Insufficient training time.
- Too few input features.
- Excessive regularization.
- Poor-quality training data.
ā ļø What Is Overfitting?
Overfitting occurs when a model learns not only the useful patterns but also the noise and random variations present in the training dataset. Although the model performs extremely well on training data, it performs poorly on unseen data.
Characteristics
- Very low training error.
- High testing error.
- Memorizes the training data.
- Poor generalization.
š Causes of Overfitting
- Model is excessively complex.
- Too many features.
- Insufficient training examples.
- Training for too many epochs.
- Presence of noisy training data.
š Underfitting vs Overfitting
| Aspect | Underfitting | Overfitting |
|---|---|---|
| Model Complexity | Too Simple | Too Complex |
| Training Accuracy | Low | Very High |
| Testing Accuracy | Low | Low |
| Generalization | Poor | Poor |
| Bias | High | Low |
| Variance | Low | High |
āļø Bias-Variance Tradeoff
A successful Machine Learning model balances bias and variance. High bias usually leads to underfitting, while high variance often leads to overfitting.
| Concept | Description |
|---|---|
| Bias | Error caused by overly simple assumptions. |
| Variance | Error caused by excessive sensitivity to training data. |
| Balanced Model | Achieves low bias and moderate variance. |
š Learning Curves
š ļø How to Reduce Underfitting
- Use a more powerful or complex model.
- Train for additional epochs when appropriate.
- Engineer more informative features.
- Reduce excessive regularization.
- Collect higher-quality training data.
š”ļø How to Prevent Overfitting
- Collect more training data.
- Use feature selection to remove irrelevant features.
- Apply regularization techniques.
- Use cross-validation.
- Apply early stopping during training.
- Use dropout in deep learning models.
- Reduce model complexity when appropriate.
š Regularization Techniques
| Technique | Purpose |
|---|---|
| L1 Regularization (Lasso) | Reduces less important feature weights to zero. |
| L2 Regularization (Ridge) | Penalizes large model weights. |
| Dropout | Temporarily disables neurons during training. |
| Early Stopping | Stops training before overfitting occurs. |
š Generalization Error
The objective of Machine Learning is to minimize the difference between training performance and performance on unseen data.
A smaller generalization error generally indicates that the model performs consistently on both training and unseen datasets.
āļø Detecting Overfitting and Underfitting
Both training accuracy and testing accuracy remain low, indicating that the model has not learned the underlying patterns effectively.
Training accuracy is very high while testing accuracy is significantly lower, indicating that the model has memorized the training data.
Training and testing accuracies are both high and close to each other, indicating strong generalization.
š» Example: Detecting Overfitting
The following example compares training and testing accuracy using a Decision Tree classifier.
overfitting_example.py
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
X = [[1], [2], [3], [4], [5], [6], [7], [8]]
y = [0, 0, 0, 1, 1, 1, 1, 1]
X_train, X_test, y_train, y_test = train_test_split(
X,
y,
test_size=0.25,
random_state=42
)
model = DecisionTreeClassifier()
model.fit(X_train, y_train)
train_accuracy = accuracy_score(
y_train,
model.predict(X_train)
)
test_accuracy = accuracy_score(
y_test,
model.predict(X_test)
)
print("Training Accuracy:", train_accuracy)
print("Testing Accuracy:", test_accuracy)š» Example: Cross-Validation
Cross-validation provides a more reliable estimate of model performance by training and evaluating the model multiple times on different subsets of the data.
cross_validation.py
from sklearn.model_selection import cross_val_score
from sklearn.tree import DecisionTreeClassifier
model = DecisionTreeClassifier()
scores = cross_val_score(
model,
X,
y,
cv=5
)
print("Cross Validation Scores:", scores)
print("Average Score:", scores.mean())š Real-World Examples
- š„ Disease prediction models that memorize patient records may overfit.
- š§ Spam filters trained on too little data may underfit.
- š Recommendation systems can overfit to historical customer behavior.
- š Autonomous driving models require balanced learning for reliable performance.
- š³ Fraud detection systems benefit from cross-validation to improve generalization.
ā Best Practices
- Use representative and diverse training datasets.
- Separate training, validation, and testing datasets.
- Monitor both training and validation performance.
- Apply feature engineering and feature selection appropriately.
- Use regularization and early stopping when needed.
- Evaluate models using cross-validation.
- Retrain models periodically with updated data.
ā ļø Common Mistakes
- Using an overly complex model for a simple problem.
- Training on insufficient or low-quality data.
- Ignoring validation performance during training.
- Using the test dataset repeatedly for model tuning.
- Assuming high training accuracy guarantees good real-world performance.
š Additional Resources
Learn more from the official Scikit-learn Cross-Validation Documentation, the Scikit-learn Model Evaluation Documentation, and the Google Machine Learning Crash Course.