📘 Introduction

Model evaluation is the process of measuring how well a machine learning model performs on unseen data. It helps determine whether a model makes accurate predictions, generalizes well to new data, and is suitable for real-world applications. Proper evaluation ensures that a model is reliable and not simply memorizing the training data.

🎯 Why Evaluate Machine Learning Models?

📚 Model Evaluation Workflow

✂️ Splitting Data

A dataset is typically divided into training and testing sets. The model learns from the training data and is evaluated using the testing data.

library(caret)

set.seed(123)

trainIndex <- createDataPartition(
  iris$Species,
  p = 0.8,
  list = FALSE
)

trainData <- iris[
  trainIndex,
]

testData <- iris[
  -trainIndex,
]

📝 Training a Classification Model

library(randomForest)

model <- randomForest(
  Species ~ .,
  data = trainData
)

🎯 Making Predictions

predictions <- predict(
  model,
  testData
)

head(predictions)

📊 Confusion Matrix

A confusion matrix summarizes classification results by comparing predicted and actual values.

library(caret)

confusionMatrix(
  predictions,
  testData$Species
)

📋 Confusion Matrix Components

📈 Classification Metrics

📊 Calculating Accuracy

actual <- c(
  "Yes",
  "No",
  "Yes",
  "Yes",
  "No"
)

predicted <- c(
  "Yes",
  "No",
  "No",
  "Yes",
  "No"
)

accuracy <- mean(
  actual == predicted
)

print(accuracy)

Output

[1] 0.8

📉 Regression Evaluation Metrics

Regression models predict continuous values and require different evaluation metrics.

📏 Mean Absolute Error (MAE)

actual <- c(
  100,
  120,
  130,
  150
)

predicted <- c(
  98,
  125,
  128,
  148
)

mae <- mean(
  abs(
    actual - predicted
  )
)

print(mae)

📐 Mean Squared Error (MSE)

mse <- mean(
  (
    actual - predicted
  )^2
)

print(mse)

📏 Root Mean Squared Error (RMSE)

rmse <- sqrt(mse)

print(rmse)

📈 Coefficient of Determination (R²)

R² measures how much of the variation in the target variable is explained by the model.

model <- lm(
  Sepal.Length ~
    Sepal.Width +
    Petal.Length +
    Petal.Width,
  data = iris
)

summary(model)

🔄 Cross-Validation

Cross-validation evaluates a model by repeatedly splitting the data into training and validation sets.

library(caret)

control <- trainControl(
  method = "cv",
  number = 10
)

model <- train(
  Species ~ .,
  data = iris,
  method = "rf",
  trControl = control
)

print(model)

📊 k-Fold Cross-Validation

In k-fold cross-validation, the dataset is divided into k equal parts. Each part serves as the validation set once while the remaining parts are used for training.

📉 Overfitting vs Underfitting

📊 ROC Curve and AUC

The Receiver Operating Characteristic (ROC) curve evaluates binary classification models across different decision thresholds. The Area Under the Curve (AUC) summarizes overall classification performance.

🌍 Real-World Example

A hospital develops a machine learning model to predict whether patients have a particular disease. The model is evaluated using a confusion matrix, accuracy, precision, recall, and cross-validation before being deployed for clinical decision support.

library(caret)

prediction <- predict(
  model,
  testData
)

evaluation <- confusionMatrix(
  prediction,
  testData$Species
)

print(evaluation)

🔄 Model Evaluation Workflow

📋 Common Evaluation Functions

⚠️ Common Mistakes

💡 Best Practices

📝 Summary

Model evaluation measures how effectively a machine learning model performs on unseen data. In this chapter, you learned how to split datasets, make predictions, evaluate classification models using confusion matrices, accuracy, precision, recall, F1 score, and specificity, assess regression models using MAE, MSE, RMSE, and R², perform cross-validation, and identify overfitting and underfitting. Mastering these evaluation techniques enables you to compare models objectively and select reliable solutions for practical machine learning tasks.