Training, Validation, and Testing

📖 Introduction

Training, validation, and testing are three essential stages in the Machine Learning workflow. Instead of using the same data for every purpose, a dataset is divided into separate subsets so that the model can learn, be optimized, and finally be evaluated on unseen data. This process helps build models that generalize well rather than simply memorizing the training examples.

Information

Separating data into training, validation, and testing datasets helps prevent overfitting and provides an unbiased estimate of model performance.

🌟 Overview

Dataset
Training Set
Validation Set
Test Set
Learn Patterns
Build Model
Hyperparameter Tuning
Model Selection
Final Evaluation
Performance Measurement

📚 Why Split the Dataset?

If the same data is used for both learning and evaluation, the model may appear to perform well simply because it has memorized the training examples. Splitting the dataset allows us to measure how effectively the model performs on new, unseen data.

  • Improves model reliability.
  • Detects overfitting and underfitting.
  • Supports fair model comparison.
  • Provides an unbiased estimate of performance.

🎯 Training Dataset

The training dataset is the largest portion of the available data. It is used to teach the Machine Learning model by allowing it to learn relationships between input features and target labels.

Purpose

  • Learn patterns from data.
  • Estimate model parameters.
  • Build the predictive model.

Typical Size

The training dataset usually contains 70–80% of the complete dataset.

⚙️ Validation Dataset

The validation dataset is used during model development to evaluate different model configurations and tune hyperparameters without exposing the model to the final testing data.

Purpose

  • Select the best-performing model.
  • Tune hyperparameters.
  • Reduce overfitting.

Typical Size

The validation dataset typically contains 10–15% of the complete dataset.

🧪 Testing Dataset

The testing dataset is reserved until the end of model development. It provides an unbiased evaluation of the model's ability to make predictions on previously unseen data.

Purpose

  • Measure final model performance.
  • Estimate real-world prediction accuracy.
  • Compare different Machine Learning models fairly.

Typical Size

The testing dataset usually represents 10–20% of the complete dataset.

📊 Dataset Split Comparison

DatasetPurposeTypical Percentage
Training SetLearn patterns and train the model.70–80%
Validation SetTune hyperparameters and select the best model.10–15%
Test SetEvaluate final model performance.10–20%

🔄 Complete Workflow

📈 Visual Representation

Complete Dataset
Training (70–80%)
Validation (10–15%)
Testing (10–20%)
Model Learning
Model Tuning
Final Evaluation

⚠️ Overfitting and Underfitting

An overfitted model memorizes the training data instead of learning general patterns. It performs well on the training dataset but poorly on unseen data.

An underfitted model is too simple to capture important relationships in the data, leading to poor performance on both training and testing datasets.

A well-generalized model performs consistently across training, validation, and testing datasets, indicating that it has learned meaningful patterns rather than memorizing data.

📐 Common Evaluation Metrics

Problem TypeCommon Metrics
ClassificationAccuracy, Precision, Recall, F1-Score
RegressionMAE, MSE, RMSE, R² Score

💻 Example: Splitting a Dataset

The following example demonstrates how to divide a dataset into training and testing subsets using scikit-learn. A validation dataset can be created by further splitting the training data if required.

train_validation_test_split.py

from sklearn.model_selection import train_test_split
import pandas as pd

data = pd.read_csv("students.csv")

X = data.drop("Result", axis=1)
y = data["Result"]

# Split into training (80%) and testing (20%)
X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.2,
    random_state=42
)

print("Training Samples:", len(X_train))
print("Testing Samples:", len(X_test))

📊 Best Practices

  • Keep the testing dataset completely unseen until final evaluation.
  • Use representative and balanced datasets whenever possible.
  • Shuffle data before splitting to reduce sampling bias.
  • Use validation data only for model selection and hyperparameter tuning.
  • Retrain the final model using the chosen configuration before deployment.

🌍 Real-World Example

  • 🏥 Train a medical diagnosis model using historical patient records.
  • 📧 Validate different spam detection models to select the best one.
  • 🛒 Test a recommendation system using new customer interactions.
  • 💳 Evaluate fraud detection models before deploying them in banking systems.
  • 🚗 Assess autonomous driving models using previously unseen driving scenarios.

📚 Additional Resources

Learn more from the official Scikit-learn Documentation, the Google Machine Learning Guides, and the TensorFlow Documentation.

Best Practice

Never evaluate a model using the same data that was used for training. Keeping the testing dataset separate ensures an honest assessment of real-world performance.

Remember

The training dataset teaches the model, the validation dataset improves the model, and the testing dataset measures how well the final model performs on unseen data.

Summary

Training, validation, and testing are fundamental stages of the Machine Learning lifecycle. The training dataset is used to learn patterns, the validation dataset is used to tune hyperparameters and select the best model, and the testing dataset provides an unbiased evaluation of final performance. Proper dataset splitting helps build models that generalize well and perform reliably in real-world applications.