Principal Component Analysis (PCA)
š Introduction
Principal Component Analysis (PCA) is one of the most widely used unsupervised dimensionality reduction techniques in machine learning. PCA transforms a dataset containing many correlated features into a smaller set of new, uncorrelated variables called Principal Components (PCs), while preserving as much of the original data's variability as possible.
Information
PCA reduces the number of features without significantly losing important information, making machine learning models faster, simpler, and less prone to overfitting.
šÆ Learning Objectives
- Understand dimensionality reduction.
- Learn the mathematical foundation of PCA.
- Understand eigenvectors and eigenvalues.
- Apply PCA for feature extraction and visualization.
š What is Dimensionality Reduction?
Dimensionality reduction is the process of reducing the number of input features while retaining the most important information. It helps simplify datasets, reduce computation time, eliminate redundancy, and improve model performance.
| High-Dimensional Data | After PCA |
|---|---|
| Many correlated features | Few independent components |
| Higher computational cost | Lower computational cost |
| Hard to visualize | Easy visualization (2D or 3D) |
š Key Concepts
| Concept | Description |
|---|---|
| Principal Component (PC) | New feature representing maximum variance. |
| Eigenvector | Direction of maximum variance. |
| Eigenvalue | Amount of variance explained by an eigenvector. |
| Covariance Matrix | Measures relationships between features. |
| Explained Variance | Percentage of total variance retained. |
š Covariance Matrix
PCA begins by computing the covariance matrix to measure how features vary together.
A high covariance indicates that two features change together, while a covariance close to zero indicates little or no linear relationship.
š Eigenvalues and Eigenvectors
PCA computes the eigenvalues and eigenvectors of the covariance matrix.
Where:
- A ā Covariance matrix.
- v ā Eigenvector (principal direction).
- Ī» ā Eigenvalue representing explained variance.
Remember
Principal components are ordered by decreasing eigenvalues. The first principal component explains the greatest amount of variance, the second explains the next greatest amount while remaining orthogonal to the first, and so on.
āļø How PCA Works
Standardize the dataset.
Compute the covariance matrix.
Calculate eigenvalues and eigenvectors.
Sort principal components by explained variance.
Select the top principal components.
Project the original data onto the selected components.
š³ PCA Workflow
Input Dataset
Standardize Features
Compute Covariance Matrix
Calculate Eigenvalues & Eigenvectors
Select Principal Components
Transform Dataset
š Explained Variance Ratio
The explained variance ratio indicates how much of the total dataset variance is retained by each principal component.
Tip
A common practice is to retain enough principal components to explain approximately 90%ā95% of the total variance.
š PCA vs Feature Selection
| Feature | PCA | Feature Selection |
|---|---|---|
| Creates New Features | Yes | No |
| Reduces Dimensionality | Yes | Yes |
| Uses Original Features | No | Yes |
| Interpretability | Lower | Higher |
š PCA vs t-SNE vs UMAP
| Feature | PCA | t-SNE | UMAP |
|---|---|---|---|
| Linear Method | Yes | No | No |
| Preserves Global Structure | Excellent | Limited | Good |
| Visualization | Good | Excellent | Excellent |
| Training Speed | Very Fast | Slow | Fast |
| Scalability | Excellent | Moderate | Excellent |
šļø Important Hyperparameters
| Hyperparameter | Description |
|---|---|
| n_components | Number of principal components. |
| svd_solver | Algorithm used for Singular Value Decomposition. |
| whiten | Scales components to unit variance. |
| random_state | Controls reproducibility (for randomized solvers). |
š Evaluation Metrics
- Explained Variance Ratio
- Cumulative Explained Variance
- Reconstruction Error
- Downstream Model Performance
āļø Advantages and Limitations
Advantages
Limitations
- Reduces dimensionality efficiently.
- Removes feature redundancy.
- Speeds up model training.
- Helps reduce overfitting.
- Excellent for visualization.
- Principal components are less interpretable than original features.
- Captures only linear relationships.
- Requires feature scaling.
- May discard useful information if too few components are retained.
š Real-World Applications
| Application | Purpose |
|---|---|
| š¼ļø Image Compression | Reduce image dimensions while preserving quality. |
| š§¬ Bioinformatics | Analyze high-dimensional gene expression data. |
| š³ Fraud Detection | Reduce redundant financial features. |
| š Finance | Summarize correlated market indicators. |
| š¤ Machine Learning | Preprocess high-dimensional datasets. |
| š Data Visualization | Project high-dimensional data into 2D or 3D. |
š» Practical Example
Principal Component Analysis Using Scikit-learn
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
import numpy as np
# Sample data
X = np.array([
[2.5, 2.4],
[0.5, 0.7],
[2.2, 2.9],
[1.9, 2.2],
[3.1, 3.0]
])
# Standardize features
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)
# Apply PCA
pca = PCA(n_components=1)
X_pca = pca.fit_transform(X_scaled)
print("Transformed Data:")
print(X_pca)
print("Explained Variance Ratio:")
print(pca.explained_variance_ratio_)ā ļø Common Mistakes
- Applying PCA without standardizing features.
- Retaining too few principal components and losing important information.
- Using PCA when feature interpretability is essential.
- Assuming PCA captures nonlinear relationships.
- Using PCA without checking the explained variance ratio.
Best Practice
Standardize numerical features before applying PCA, examine the cumulative explained variance to determine the appropriate number of principal components, retain enough components to preserve approximately 90ā95% of the variance, and use PCA primarily for dimensionality reduction, visualization, noise reduction, and preprocessing rather than direct feature interpretation.
š Summary
Summary
Principal Component Analysis (PCA) is a linear dimensionality reduction technique that transforms correlated features into a smaller set of orthogonal principal components while preserving most of the dataset's variance. By leveraging covariance matrices, eigenvectors, and eigenvalues, PCA simplifies high-dimensional datasets, reduces computational complexity, improves visualization, and often enhances machine learning performance. It remains one of the most fundamental preprocessing techniques in data science, statistics, and machine learning.