๐ Introduction
K-Means is one of the most popular unsupervised machine learning algorithms used for clustering. It groups similar data points into K distinct clusters by minimizing the distance between data points and their corresponding cluster centroids.
Mini-Batch K-Means is an optimized version of K-Means that processes small random subsets (mini-batches) of the dataset instead of the entire dataset during each iteration. This significantly reduces training time while producing clustering results similar to standard K-Means.
Information
๐ฏ Learning Objectives
- Understand clustering and unsupervised learning.
- Learn how K-Means partitions data into clusters.
- Understand the Mini-Batch K-Means optimization.
- Compare K-Means and Mini-Batch K-Means.
๐ What is Clustering?
Clustering is an unsupervised learning technique that groups similar observations together without using labeled target values.
| Supervised Learning | Unsupervised Learning |
|---|---|
| Uses labeled data. | Uses unlabeled data. |
| Predicts target values. | Discovers hidden groups or patterns. |
| Classification & Regression. | Clustering & Dimensionality Reduction. |
๐ K-Means Algorithm
K-Means partitions the dataset into K clusters by iteratively assigning each observation to the nearest centroid and then updating the centroid positions until convergence.
Objective Function
Where:
- K โ Number of clusters.
- Cแตข โ Cluster i.
- ฮผแตข โ Centroid of cluster i.
- ||x-ฮผแตข||ยฒ โ Squared Euclidean distance between a point and its centroid.
Remember
โ๏ธ How K-Means Works
Select the number of clusters (K).
Initialize K centroids randomly or using K-Means++.
Assign every observation to the nearest centroid.
Recompute centroids as the mean of assigned points.
Repeat assignment and update until centroids no longer change significantly or the maximum number of iterations is reached.
๐ณ K-Means Workflow
๐ Mini-Batch K-Means
Mini-Batch K-Means improves computational efficiency by updating centroids using randomly selected mini-batches rather than the full dataset during every iteration.
Select a random mini-batch of observations.
Assign mini-batch samples to the nearest centroids.
Update centroids using only the mini-batch.
Repeat with new mini-batches until convergence.
Tip
๐ K-Means vs Mini-Batch K-Means
| Feature | K-Means | Mini-Batch K-Means |
|---|---|---|
| Training Data | Entire dataset | Random mini-batches |
| Training Speed | Slower | Much Faster |
| Memory Usage | Higher | Lower |
| Clustering Accuracy | Highest | Very Similar |
| Best For | Small & Medium datasets | Large datasets |
๐ฏ Choosing the Number of Clusters
Elbow Method
Plot the Within-Cluster Sum of Squares (WCSS) against different values of K. The optimal number of clusters is often located at the "elbow" point where further increases in K provide diminishing improvements.
Silhouette Score
The Silhouette Score measures how similar a data point is to its own cluster compared to other clusters.
Higher values indicate better-defined clusters.
๐ Distance Measure
K-Means primarily uses Euclidean Distance to determine the nearest centroid.
๐๏ธ Important Hyperparameters
| Hyperparameter | Description |
|---|---|
| n_clusters | Number of clusters. |
| init | Centroid initialization method (e.g., K-Means++). |
| max_iter | Maximum number of iterations. |
| batch_size | Mini-batch size (Mini-Batch K-Means). |
| random_state | Controls reproducibility. |
๐ Evaluation Metrics
- Within-Cluster Sum of Squares (WCSS / Inertia)
- Silhouette Score
- Davies-Bouldin Index
- Calinski-Harabasz Index
โ๏ธ Advantages and Limitations
- Simple and easy to implement.
- Fast convergence.
- Efficient for large numerical datasets.
- Mini-Batch K-Means scales to massive datasets.
- Works well for compact, spherical clusters.
- Requires the number of clusters to be specified beforehand.
- Sensitive to centroid initialization.
- Assumes clusters are roughly spherical and similar in size.
- Sensitive to outliers.
- Primarily designed for numerical features.
๐ Real-World Applications
| Application | Purpose |
|---|---|
| ๐ Customer Segmentation | Group customers based on purchasing behavior. |
| ๐ผ๏ธ Image Compression | Reduce the number of image colors. |
| ๐ Document Clustering | Organize similar documents. |
| ๐งฌ Bioinformatics | Cluster genes with similar expression patterns. |
| ๐ก Network Traffic Analysis | Identify usage patterns and anomalies. |
| ๐ Market Research | Discover hidden consumer groups. |
๐ป Practical Example
K-Means and Mini-Batch K-Means Using Scikit-learn
from sklearn.cluster import KMeans
from sklearn.cluster import MiniBatchKMeans
import numpy as np
# Sample data
X = np.array([
[1, 2], [1, 3], [2, 2],
[8, 8], [9, 8], [8, 9]
])
# K-Means
kmeans = KMeans(
n_clusters=2,
random_state=42
)
kmeans.fit(X)
# Mini-Batch K-Means
mini = MiniBatchKMeans(
n_clusters=2,
batch_size=3,
random_state=42
)
mini.fit(X)
print("K-Means Labels:", kmeans.labels_)
print("Mini-Batch Labels:", mini.labels_)
print("Cluster Centers:")
print(kmeans.cluster_centers_)โ ๏ธ Common Mistakes
- Choosing an inappropriate value for K.
- Ignoring feature scaling before clustering.
- Using K-Means on categorical features without suitable encoding.
- Assuming clusters of arbitrary shapes can be identified accurately.
- Ignoring the impact of outliers on centroid positions.