Machine Learning Algorithms

Linear Discriminant Analysis (LDA)

📖 Introduction

Linear Discriminant Analysis (LDA) is a supervised dimensionality reduction and classification algorithm that projects data onto a lower-dimensional space while maximizing the separation between different classes. Unlike Principal Component Analysis (PCA), which focuses on preserving variance, LDA focuses on improving class separability.

Information

LDA uses class labels during training, making it a supervised learning technique. It is commonly used as both a preprocessing method for dimensionality reduction and as a classifier.

🎯 Learning Objectives

Understand Linear Discriminant Analysis.
Learn the difference between PCA and LDA.
Understand within-class and between-class scatter.
Apply LDA for dimensionality reduction and classification.

🌟 What is Linear Discriminant Analysis?

Linear Discriminant Analysis transforms the original feature space into a lower-dimensional space where observations belonging to different classes become as distinct as possible. It achieves this by maximizing the distance between class means while minimizing the spread of observations within each class.

Characteristic	Linear Discriminant Analysis
Learning Type	Supervised
Main Objective	Maximize Class Separation
Uses Class Labels	Yes
Applications	Classification & Dimensionality Reduction

📍 Key Concepts

Concept	Description
Class Mean	Average feature vector for each class.
Within-Class Scatter	Variation among observations within the same class.
Between-Class Scatter	Variation between different class means.
Linear Discriminant	Projection direction that maximizes class separation.
Discriminant Components	New transformed features obtained after projection.

📊 Within-Class Scatter Matrix

The within-class scatter matrix measures the spread of observations inside each class.

S_{W} = i = 1 \sum c x \in C_{i} \sum (x - μ_{i}) (x - μ_{i})^{T}

Where:

Cᵢ — Class i.
μᵢ — Mean vector of class i.
c — Number of classes.

📈 Between-Class Scatter Matrix

The between-class scatter matrix measures how far apart different class means are.

S_{B} = i = 1 \sum c N_{i} (μ_{i} - μ) (μ_{i} - μ)^{T}

Where:

Nᵢ — Number of observations in class i.
μ — Overall dataset mean.

Remember

Good class separation is achieved by maximizing the between-class scatter while minimizing the within-class scatter.

🎯 LDA Optimization Objective

LDA finds projection directions that maximize the ratio of between-class scatter to within-class scatter.

J (w) = \frac{w ^{T} S _{B} w}{w ^{T} S _{W} w}

The optimal projection vector maximizes this objective function.

⚙️ How LDA Works

📥

Compute the mean vector for each class.

📊

Calculate the within-class scatter matrix.

📈

Compute the between-class scatter matrix.

🔢

Solve the generalized eigenvalue problem.

✂️

Select the discriminant vectors with the largest eigenvalues.

🎯

Project the original data onto the new discriminant space.

🌳 LDA Workflow

Input Dataset with Labels

Compute Class Means

Calculate Scatter Matrices

Solve Eigenvalue Problem

Select Linear Discriminants

Transform Dataset

📊 PCA vs LDA

Feature	PCA	LDA
Learning Type	Unsupervised	Supervised
Uses Class Labels	No	Yes
Main Objective	Maximize Variance	Maximize Class Separation
Output Components	Principal Components	Linear Discriminants
Typical Use	Feature Extraction	Classification & Feature Extraction

📊 Number of Components

Unlike PCA, the maximum number of linear discriminants is limited by the number of classes.

M a x im u m C o m p o n e n t s = C - 1

Where:

C — Number of classes.

Tip

For a dataset containing three classes, LDA can produce at most two discriminant components.

🎛️ Important Hyperparameters

Hyperparameter	Description
solver	Algorithm used to compute discriminants.
n_components	Number of discriminant components.
shrinkage	Regularization for covariance estimation.
store_covariance	Stores covariance matrix after training.

📊 Evaluation Metrics

Classification

Dimensionality Reduction

Accuracy
Precision
Recall
F1-Score
ROC-AUC
Confusion Matrix

Class Separability
Visualization Quality
Downstream Model Performance

⚖️ Advantages and Limitations

Advantages

Limitations

Excellent class separation.
Reduces dimensionality while preserving discriminative information.
Simple and computationally efficient.
Works well when class distributions are approximately Gaussian.
Improves many classification algorithms.

Requires labeled training data.
Assumes classes have similar covariance matrices.
Assumes approximately Gaussian class distributions.
Maximum components are limited to C − 1.
Less effective when assumptions are strongly violated.

🌍 Real-World Applications

Application	Purpose
🩺 Medical Diagnosis	Separate patients into diagnostic groups.
😊 Face Recognition	Reduce image dimensions while preserving identity information.
📧 Spam Detection	Improve document classification.
💳 Credit Risk Analysis	Classify loan applicants.
🧬 Bioinformatics	Classify biological samples.
🎤 Speech Recognition	Extract discriminative acoustic features.

💻 Practical Example

Linear Discriminant Analysis Using Scikit-learn

from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
from sklearn.datasets import load_iris

# Load dataset
X, y = load_iris(return_X_y=True)

# Create LDA model
lda = LinearDiscriminantAnalysis(
    n_components=2
)

# Transform data
X_lda = lda.fit_transform(X, y)

print("Transformed Shape:", X_lda.shape)

# Classification
predictions = lda.predict(X)

print("Predictions:")
print(predictions[:10])

⚠️ Common Mistakes

Using LDA without labeled training data.
Confusing Linear Discriminant Analysis with Latent Dirichlet Allocation (both abbreviated as LDA).
Ignoring violations of Gaussian distribution assumptions.
Expecting more than C − 1 discriminant components.
Applying LDA when classes are highly nonlinear and not linearly separable.

Best Practice

Standardize numerical features when appropriate, verify that class distributions are reasonably Gaussian with similar covariance structures, choose n_components ≤ C − 1, and use LDA as both a dimensionality reduction technique and a classifier when maximizing class separation is more important than preserving overall variance.

📚 Summary

Summary

Linear Discriminant Analysis (LDA) is a supervised dimensionality reduction and classification technique that projects data into a lower-dimensional space by maximizing the separation between classes. Unlike PCA, which preserves variance without using labels, LDA incorporates class information to produce highly discriminative features. Because of its efficiency, interpretability, and strong classification performance under appropriate assumptions, LDA is widely used in medical diagnosis, face recognition, bioinformatics, finance, and speech recognition.

Menu

Machine Learning Algorithms

📖 Introduction

Information

🎯 Learning Objectives

🌟 What is Linear Discriminant Analysis?

📍 Key Concepts

📊 Within-Class Scatter Matrix

📈 Between-Class Scatter Matrix

Remember

🎯 LDA Optimization Objective

⚙️ How LDA Works

🌳 LDA Workflow

📊 PCA vs LDA

📊 Number of Components

Tip

🎛️ Important Hyperparameters

📊 Evaluation Metrics

⚖️ Advantages and Limitations

🌍 Real-World Applications

💻 Practical Example

Linear Discriminant Analysis Using Scikit-learn

⚠️ Common Mistakes

Best Practice

📚 Summary

Summary

🔗 Further Reading

📖 Introduction

ℹ️Information

🎯 Learning Objectives

🌟 What is Linear Discriminant Analysis?

📍 Key Concepts

📊 Within-Class Scatter Matrix

📈 Between-Class Scatter Matrix

🧠Remember

🎯 LDA Optimization Objective

⚙️ How LDA Works

🌳 LDA Workflow

📊 PCA vs LDA

📊 Number of Components

💡Tip

🎛️ Important Hyperparameters

📊 Evaluation Metrics

⚖️ Advantages and Limitations

🌍 Real-World Applications

💻 Practical Example

Linear Discriminant Analysis Using Scikit-learn

⚠️ Common Mistakes

⭐Best Practice

📚 Summary

📋Summary

🔗 Further Reading

Information

Remember

Tip

Best Practice

Summary