Anomaly Detection Algorithms
š Introduction
Anomaly Detection (also known as Outlier Detection) is a machine learning technique used to identify observations that significantly deviate from the normal behavior of a dataset. These unusual observations, called anomalies or outliers, may indicate fraud, equipment failure, cyberattacks, medical abnormalities, or data quality issues.
Information
Anomaly Detection is widely used in fraud detection, cybersecurity, predictive maintenance, healthcare, finance, manufacturing, and network monitoring to identify rare but important events.
šÆ Learning Objectives
- Understand anomaly detection and outlier detection.
- Learn different categories of anomaly detection.
- Understand major anomaly detection algorithms.
- Compare Isolation Forest, One-Class SVM, Local Outlier Factor, and Elliptic Envelope.
š What is Anomaly Detection?
Anomaly Detection aims to distinguish normal observations from abnormal ones. Since anomalies are rare and often unknown beforehand, many anomaly detection methods are unsupervised or semi-supervised.
| Characteristic | Anomaly Detection |
|---|---|
| Learning Type | Unsupervised, Semi-Supervised & Supervised |
| Main Goal | Detect Unusual Observations |
| Output | Normal or Anomaly |
| Typical Dataset | Highly Imbalanced |
š Types of Anomalies
| Type | Description | Example |
|---|---|---|
| Point Anomaly | Single abnormal observation. | Credit card fraud transaction. |
| Contextual Anomaly | Observation abnormal only in a specific context. | High temperature during winter. |
| Collective Anomaly | Group of observations forming an unusual pattern. | Network attack traffic. |
š Key Concepts
| Concept | Description |
|---|---|
| Outlier | Observation significantly different from others. |
| Normal Data | Majority of observations following expected patterns. |
| Anomaly Score | Numerical measure indicating abnormality. |
| Contamination | Estimated proportion of anomalies in the dataset. |
| Decision Boundary | Separates normal observations from anomalies. |
š² Isolation Forest
Isolation Forest detects anomalies by recursively partitioning the data using randomly selected features and split values. Since anomalies are few and different, they are isolated using fewer splits than normal observations.
Dataset
Random Feature Selection
Random Split
Build Isolation Trees
Anomaly Score
Remember
Anomalies generally have shorter average path lengths in Isolation Trees because they are easier to isolate.
Anomaly Score
Where:
- h(x) ā Path length for observation x.
- c(n) ā Average path length of a binary search tree.
šÆ One-Class SVM
One-Class Support Vector Machine (One-Class SVM) learns a boundary enclosing normal observations and classifies observations outside this boundary as anomalies.
Normal Training Data
Learn Decision Boundary
New Observation
Inside Boundary ā Normal
Outside Boundary ā Anomaly
š Local Outlier Factor (LOF)
Local Outlier Factor (LOF) identifies anomalies by comparing the local density of an observation with the densities of its neighboring observations.
| LOF Score | Interpretation |
|---|---|
| ā 1 | Normal observation. |
| > 1 | Potential anomaly. |
š Elliptic Envelope
Elliptic Envelope assumes the data follows a Gaussian distribution and fits an ellipse around normal observations using robust covariance estimation.
Input Data
Estimate Covariance
Fit Ellipse
Outside Ellipse ā Anomaly
Tip
Elliptic Envelope performs best when numerical features approximately follow a multivariate Gaussian distribution.
āļø General Anomaly Detection Workflow
Collect and preprocess data.
Scale numerical features if required.
Train the anomaly detection algorithm.
Compute anomaly scores.
Identify observations exceeding the anomaly threshold.
Investigate detected anomalies.
š³ Anomaly Detection Pipeline
Input Dataset
Data Preprocessing
Train Detection Model
Compute Anomaly Scores
Threshold Selection
Normal / Anomaly
š Algorithm Comparison
| Feature | Isolation Forest | One-Class SVM | LOF | Elliptic Envelope |
|---|---|---|---|---|
| Learning Type | Unsupervised | Semi-Supervised | Unsupervised | Unsupervised |
| Scalability | Excellent | Moderate | Moderate | Good |
| Handles High Dimensions | Excellent | Good | Moderate | Limited |
| Distribution Assumption | No | No | No | Gaussian |
| Best For | Large Datasets | Known Normal Data | Local Outliers | Gaussian Data |
šļø Important Hyperparameters
| Algorithm | Important Hyperparameters |
|---|---|
| Isolation Forest | n_estimators, max_samples, contamination |
| One-Class SVM | kernel, gamma, nu |
| LOF | n_neighbors, metric, contamination |
| Elliptic Envelope | contamination, support_fraction |
š Evaluation Metrics
Labeled Data
Unlabeled Data
- Precision
- Recall
- F1-Score
- ROC-AUC
- Precision-Recall Curve
- Confusion Matrix
- Silhouette Analysis (limited scenarios)
- Manual Investigation
- Business Validation
- Anomaly Score Distribution
āļø Advantages and Limitations
Advantages
Limitations
- Detects rare and unusual events.
- Often requires little or no labeled data.
- Applicable across many industries.
- Supports fraud prevention and predictive maintenance.
- Many algorithms scale efficiently to large datasets.
- Anomalies are often difficult to define.
- Datasets are usually highly imbalanced.
- Selecting an appropriate anomaly threshold is challenging.
- High false-positive rates may require manual review.
- Performance depends on feature quality and preprocessing.
š Real-World Applications
| Application | Purpose |
|---|---|
| š³ Fraud Detection | Identify suspicious financial transactions. |
| š”ļø Cybersecurity | Detect network intrusions and malware. |
| š Predictive Maintenance | Detect abnormal equipment behavior before failure. |
| š„ Healthcare | Identify unusual patient conditions or medical signals. |
| š Financial Markets | Detect unusual trading activities. |
| š” IoT Monitoring | Detect abnormal sensor readings. |
š» Practical Example
Isolation Forest Using Scikit-learn
from sklearn.ensemble import IsolationForest
from sklearn.datasets import make_blobs
import matplotlib.pyplot as plt
# Generate sample data
X, _ = make_blobs(
n_samples=300,
centers=1,
cluster_std=0.6,
random_state=42
)
# Train Isolation Forest
model = IsolationForest(
contamination=0.05,
random_state=42
)
labels = model.fit_predict(X)
# -1 = anomaly, 1 = normal
print(labels[:20])
plt.scatter(
X[:,0],
X[:,1],
c=labels,
cmap="coolwarm"
)
plt.title("Isolation Forest Anomaly Detection")
plt.show()ā ļø Common Mistakes
- Ignoring feature scaling for distance-based algorithms such as LOF and One-Class SVM.
- Using an incorrect contamination value.
- Assuming every detected anomaly is an error or fraud.
- Evaluating models only by accuracy on highly imbalanced datasets.
- Skipping domain knowledge when validating detected anomalies.
Best Practice
Normalize numerical features when appropriate, estimate the contamination parameter carefully, use Isolation Forest for large and high-dimensional datasets, prefer LOF when detecting local density anomalies, apply One-Class SVM when reliable normal training data is available, and always validate detected anomalies using business or domain expertise before taking action.
š Summary
Summary
Anomaly Detection identifies rare observations that differ significantly from normal behavior. Popular algorithms include Isolation Forest, which isolates anomalies using random trees, One-Class SVM, which learns a boundary around normal data, Local Outlier Factor (LOF), which compares local densities, and Elliptic Envelope, which models Gaussian-distributed data. These techniques play a critical role in fraud detection, cybersecurity, predictive maintenance, healthcare, IoT monitoring, and many other domains where identifying unusual behavior is essential.