Machine Learning Algorithms

Manifold Learning (t-SNE & UMAP)

📖 Introduction

Manifold Learning is a family of unsupervised nonlinear dimensionality reduction techniques that assume high-dimensional data lies on a lower-dimensional manifold embedded within a higher-dimensional space. Instead of preserving only linear relationships like PCA, manifold learning captures complex nonlinear structures while maintaining meaningful relationships among observations.

Two of the most popular manifold learning algorithms are t-Distributed Stochastic Neighbor Embedding (t-SNE) and Uniform Manifold Approximation and Projection (UMAP), both widely used for data visualization and exploratory analysis.

Information

Manifold learning is primarily used for visualization, exploratory data analysis, and discovering hidden nonlinear structures in high-dimensional datasets.

🎯 Learning Objectives

Understand manifold learning.
Learn how t-SNE preserves local neighborhoods.
Understand how UMAP models manifold structures.
Compare PCA, t-SNE, and UMAP.

🌟 What is Manifold Learning?

Manifold learning assumes that although data may contain hundreds or thousands of features, its true intrinsic structure can often be represented using far fewer dimensions. These algorithms attempt to uncover this hidden manifold while preserving neighborhood relationships.

Characteristic	Manifold Learning
Learning Type	Unsupervised
Relationship Modeled	Nonlinear
Main Purpose	Visualization & Feature Extraction
Typical Output	2D or 3D Embedding

📍 Key Concepts

Concept	Description
Manifold	Lower-dimensional structure hidden inside high-dimensional data.
Embedding	Low-dimensional representation of the original data.
Local Neighborhood	Nearby observations that should remain close after projection.
Global Structure	Overall arrangement of clusters.
Similarity Graph	Graph describing relationships among observations.

🌐 t-Distributed Stochastic Neighbor Embedding (t-SNE)

t-SNE is a nonlinear dimensionality reduction algorithm designed primarily for visualization. It converts pairwise similarities between observations into probability distributions and attempts to preserve these neighborhood relationships in a lower-dimensional space.

Key Characteristics

Excellent at preserving local neighborhoods.
Produces visually separated clusters.
Widely used for visualization of high-dimensional data.
Computationally intensive for large datasets.

High-Dimensional Similarity

p_{ij} = \frac{exp ( - \frac{∣∣ x _{i} - x _{j} ∣ ∣ ^{2}}{2 σ _{i}^{2}} )}{\sum _{k \neq = l} exp ( ... )}

Nearby observations have higher probabilities of being neighbors.

Low-Dimensional Similarity

q_{ij} = \frac{( 1 + ∣∣ y _{i} - y _{j} ∣ ∣ ^{2} ) ^{- 1}}{\sum ( 1 + ∣∣ y _{k} - y _{l} ∣ ∣ ^{2} ) ^{- 1}}

Remember

t-SNE uses a Student's t-distribution in the low-dimensional space to reduce the "crowding problem" and separate clusters more effectively.

⚙️ How t-SNE Works

📥

Compute pairwise similarities in the original feature space.

📊

Initialize low-dimensional embeddings randomly.

🔄

Compute similarities in the embedding space.

📉

Minimize the divergence between both probability distributions.

🎯

Obtain the final low-dimensional visualization.

🌐 Uniform Manifold Approximation and Projection (UMAP)

UMAP is a modern manifold learning algorithm that constructs a graph representation of the data and optimizes a low-dimensional embedding while preserving both local and much of the global structure.

Key Characteristics

Faster than t-SNE.
Scales well to large datasets.
Preserves local and global structures.
Supports downstream machine learning tasks.

UMAP Workflow

📥

Construct a k-nearest neighbor graph.

🕸️

Estimate the manifold structure.

📉

Optimize a low-dimensional embedding.

🎯

Produce the final visualization.

Tip

UMAP generally preserves more of the global data structure than t-SNE while being significantly faster on large datasets.

🌳 Manifold Learning Workflow

High-Dimensional Dataset

Compute Neighborhood Relationships

Learn Manifold Structure

Optimize Low-Dimensional Embedding

2D / 3D Visualization

📊 PCA vs t-SNE vs UMAP

Feature	PCA	t-SNE	UMAP
Relationship Type	Linear	Nonlinear	Nonlinear
Visualization Quality	Good	Excellent	Excellent
Preserves Local Structure	Limited	Excellent	Excellent
Preserves Global Structure	Excellent	Limited	Good
Scalability	Excellent	Moderate	Excellent

🎛️ Important Hyperparameters

t-SNE

UMAP

Hyperparameter	Description
perplexity	Controls neighborhood size.
learning_rate	Gradient optimization step size.
n_iter	Maximum optimization iterations.
init	Embedding initialization method.

Hyperparameter	Description
n_neighbors	Controls local neighborhood size.
min_dist	Minimum distance between embedded points.
metric	Distance metric for similarity calculation.
n_components	Embedding dimensionality.

📊 Evaluation Methods

Visualization Quality
Neighborhood Preservation
Trustworthiness Score
Continuity Score
Downstream Model Performance

⚖️ Advantages and Limitations

Advantages

Limitations

Captures nonlinear structures.
Produces highly informative visualizations.
Excellent for exploratory data analysis.
UMAP scales efficiently to large datasets.
Works well for image, text, and biological data.

Embeddings may vary across runs unless a random seed is fixed.
t-SNE is computationally expensive.
Distances between clusters in t-SNE should not always be interpreted literally.
Hyperparameter tuning significantly affects results.
Primarily intended for visualization rather than predictive modeling.

🌍 Real-World Applications

Application	Purpose
🧬 Single-Cell Genomics	Visualize cell populations.
🖼️ Computer Vision	Visualize image embeddings.
🤖 Deep Learning	Inspect learned feature representations.
📄 Natural Language Processing	Visualize word and document embeddings.
🛒 Customer Analytics	Explore customer segments.
🧠 Neuroscience	Analyze neural activity patterns.

💻 Practical Example

t-SNE and UMAP Using Python

from sklearn.datasets import load_digits
from sklearn.manifold import TSNE
import umap.umap_ as umap

# Load dataset
X, y = load_digits(return_X_y=True)

# t-SNE
tsne = TSNE(
    n_components=2,
    perplexity=30,
    random_state=42
)

X_tsne = tsne.fit_transform(X)

# UMAP
umap_model = umap.UMAP(
    n_components=2,
    n_neighbors=15,
    min_dist=0.1,
    random_state=42
)

X_umap = umap_model.fit_transform(X)

print("t-SNE Shape:", X_tsne.shape)
print("UMAP Shape:", X_umap.shape)

⚠️ Common Mistakes

Interpreting distances between distant t-SNE clusters as meaningful.
Using manifold learning as a replacement for feature engineering in predictive models.
Ignoring feature scaling before applying the algorithms.
Using default hyperparameters without experimentation.
Comparing embeddings generated with different random seeds without caution.

Best Practice

Standardize numerical features before applying manifold learning, use PCA as a preprocessing step for very high-dimensional datasets to improve speed and reduce noise, choose t-SNE for high-quality visualizations of smaller datasets, prefer UMAP for larger datasets and when preserving both local and global structures is important, and fix the random seed to improve reproducibility.

📚 Summary

Summary

Manifold Learning uncovers nonlinear structures hidden within high-dimensional datasets by projecting them into lower-dimensional spaces while preserving neighborhood relationships. t-SNE excels at producing visually distinct clusters through excellent local neighborhood preservation, making it ideal for visualization. UMAP extends these capabilities by providing faster computation, improved scalability, and better preservation of global structure. Together, these techniques have become indispensable tools for exploratory data analysis, visualization, genomics, computer vision, natural language processing, and modern machine learning research.

Menu

Machine Learning Algorithms

📖 Introduction

Information

🎯 Learning Objectives

🌟 What is Manifold Learning?

📍 Key Concepts

🌐 t-Distributed Stochastic Neighbor Embedding (t-SNE)

Key Characteristics

High-Dimensional Similarity

Low-Dimensional Similarity

Remember

⚙️ How t-SNE Works

🌐 Uniform Manifold Approximation and Projection (UMAP)

Key Characteristics

UMAP Workflow

Tip

🌳 Manifold Learning Workflow

📊 PCA vs t-SNE vs UMAP

🎛️ Important Hyperparameters

📊 Evaluation Methods

⚖️ Advantages and Limitations

🌍 Real-World Applications

💻 Practical Example

t-SNE and UMAP Using Python

⚠️ Common Mistakes

Best Practice

📚 Summary

Summary

🔗 Further Reading

📖 Introduction

ℹ️Information

🎯 Learning Objectives

🌟 What is Manifold Learning?

📍 Key Concepts

🌐 t-Distributed Stochastic Neighbor Embedding (t-SNE)

Key Characteristics

High-Dimensional Similarity

Low-Dimensional Similarity

🧠Remember

⚙️ How t-SNE Works

🌐 Uniform Manifold Approximation and Projection (UMAP)

Key Characteristics

UMAP Workflow

💡Tip

🌳 Manifold Learning Workflow

📊 PCA vs t-SNE vs UMAP

🎛️ Important Hyperparameters

📊 Evaluation Methods

⚖️ Advantages and Limitations

🌍 Real-World Applications

💻 Practical Example

t-SNE and UMAP Using Python

⚠️ Common Mistakes

⭐Best Practice

📚 Summary

📋Summary

🔗 Further Reading

Information

Remember

Tip

Best Practice

Summary