K-Nearest Neighbors (KNN)
📖 Introduction
K-Nearest Neighbors (KNN) is a simple and intuitive supervised machine learning algorithm used for both classification and regression. Instead of learning an explicit mathematical model during training, KNN makes predictions by finding the K most similar training samples to a new observation using a distance metric.
Information
KNN is known as a lazy learning or instance-based learning algorithm because it stores the training data and performs computation only when making predictions.
🎯 Objectives of KNN
- Classify data into predefined categories.
- Predict continuous numerical values.
- Identify similar observations.
- Provide a simple baseline for supervised learning tasks.
- Capture nonlinear decision boundaries.
🧠 Key Concepts
| Concept | Description |
|---|---|
| K | Number of nearest neighbors used for prediction. |
| Distance Metric | Measures similarity between data points. |
| Majority Voting | Classification prediction based on the most common class. |
| Average Value | Regression prediction based on the average of neighbors. |
📊 How KNN Works
Store the labeled training dataset.
Receive a new data point for prediction.
Calculate the distance from the new point to every training sample.
Select the K nearest neighbors.
Use majority voting (classification) or averaging (regression).
Return the final prediction.
📐 Distance Metrics
Euclidean
Manhattan
Minkowski
Cosine
Euclidean Distance
Most commonly used for continuous numerical features.
Manhattan Distance
Suitable for grid-like paths and datasets with many dimensions.
Minkowski Distance
Generalizes Euclidean and Manhattan distances.
Cosine Distance
Based on the angle between vectors rather than absolute distance. It is commonly used for text and high-dimensional feature spaces.
🎛️ Choosing the Value of K
| Value of K | Effect |
|---|---|
| Small K (1–3) | Flexible model, higher variance, sensitive to noise. |
| Moderate K (5–11) | Balanced bias and variance. |
| Large K | Smoother decision boundary but may underfit. |
Remember
There is no universally optimal value of K. It is typically selected using cross-validation.
⚙️ KNN Workflow
Collect Dataset
Clean & Scale Features
Store Training Data
Compute Distances
Find K Nearest Neighbors
Predict Output
📊 Classification vs Regression
| Aspect | KNN Classification | KNN Regression |
|---|---|---|
| Output | Class Label | Continuous Value |
| Prediction Method | Majority Voting | Average of Neighbors |
| Common Applications | Spam Detection, Image Recognition | House Price Prediction |
📈 Evaluation Metrics
Classification
Regression
- Accuracy
- Precision
- Recall
- F1-Score
- ROC-AUC
- Mean Absolute Error (MAE)
- Mean Squared Error (MSE)
- Root Mean Squared Error (RMSE)
- R² Score
⚖️ Advantages and Limitations
Advantages
Limitations
- Simple and easy to understand.
- No explicit training phase.
- Works for both classification and regression.
- Can model nonlinear relationships.
- Naturally adapts to new training samples.
- Prediction becomes slow for large datasets.
- Requires substantial memory.
- Sensitive to irrelevant features.
- Feature scaling is essential.
- Performance depends heavily on the choice of K.
🌍 Real-World Applications
| Application | Purpose |
|---|---|
| 🖼️ Image Classification | Recognize similar images. |
| 🎬 Recommendation Systems | Recommend similar products or movies. |
| 🏥 Medical Diagnosis | Classify diseases using patient similarity. |
| 🏠 House Price Prediction | Estimate prices using nearby properties. |
| 💳 Fraud Detection | Identify unusual transactions. |
| 🌱 Pattern Recognition | Identify similar observations. |
💻 Practical Example
K-Nearest Neighbors Classification Using Scikit-learn
from sklearn.neighbors import KNeighborsClassifier
import numpy as np
# Sample data
X = np.array([[1], [2], [3], [4], [5], [6]])
y = np.array([0, 0, 0, 1, 1, 1])
# Create KNN model
model = KNeighborsClassifier(n_neighbors=3)
# Train model
model.fit(X, y)
# Predict
prediction = model.predict([[3.5]])
print("Predicted Class:", prediction[0])⚠️ Common Mistakes
- Not standardizing or normalizing numerical features.
- Choosing an inappropriate value of K.
- Using KNN with very large datasets without optimization.
- Ignoring irrelevant or redundant features.
- Using Euclidean Distance for categorical variables without proper encoding.
Best Practice
Always scale numerical features before applying KNN because distance calculations are highly sensitive to feature magnitudes. Use cross-validation to select the optimal value of K and experiment with different distance metrics based on the nature of the data.
📚 Summary
Summary
K-Nearest Neighbors (KNN) is a simple yet powerful supervised learning algorithm that predicts outcomes based on the similarity between observations. It supports both classification and regression tasks without requiring an explicit training phase. While KNN performs well on small to medium-sized datasets with properly scaled features, its prediction time and memory requirements increase as the dataset grows. Selecting an appropriate value of K, choosing a suitable distance metric, and performing feature scaling are essential for achieving reliable performance.