Logistic Regression
๐ Introduction
Logistic Regression is a supervised machine learning algorithm used for classification problems. Despite its name, it is a classification algorithm, not a regression algorithm. Logistic Regression predicts the probability that an observation belongs to a particular class by using the logistic (sigmoid) function.
Information
Logistic Regression is widely used for binary classification tasks such as spam detection, disease diagnosis, fraud detection, customer churn prediction, and sentiment analysis.
๐ฏ Objectives of Logistic Regression
- Predict categorical outcomes.
- Estimate class probabilities.
- Create decision boundaries between classes.
- Provide interpretable classification models.
- Serve as a strong baseline for classification tasks.
๐ง Types of Logistic Regression
| Type | Number of Classes | Example |
|---|---|---|
| Binary Logistic Regression | 2 | Spam vs Not Spam |
| Multinomial Logistic Regression | 3 or More | Image Classification |
| Ordinal Logistic Regression | Ordered Categories | Customer Satisfaction Ratings |
๐ Mathematical Model
Logistic Regression first computes a linear combination of the input features and then transforms the result into a probability using the Sigmoid Function.
Linear Equation
Sigmoid Function
The sigmoid function converts any real-valued input into a probability between 0 and 1.
๐ Classification Decision
| Predicted Probability | Predicted Class |
|---|---|
| โฅ 0.50 | Positive Class (1) |
| < 0.50 | Negative Class (0) |
Remember
The classification threshold is commonly 0.5, but it can be adjusted depending on the application's precision and recall requirements.
โ๏ธ Logistic Regression Workflow
Collect and preprocess the dataset.
Handle missing values and encode categorical variables.
Scale numerical features if necessary.
Train the Logistic Regression model.
Predict class probabilities.
Assign class labels using a decision threshold.
๐ Cost Function
Logistic Regression uses the Binary Cross-Entropy (Log Loss) instead of Mean Squared Error because it is better suited for probability estimation.
๐ Evaluation Metrics
| Metric | Purpose |
|---|---|
| Accuracy | Overall prediction correctness. |
| Precision | Correct positive predictions. |
| Recall | Ability to identify positive cases. |
| F1-Score | Balance between precision and recall. |
| ROC-AUC | Measures classification performance across thresholds. |
| Confusion Matrix | Detailed classification summary. |
โ๏ธ Linear Regression vs Logistic Regression
| Feature | Linear Regression | Logistic Regression |
|---|---|---|
| Problem Type | Regression | Classification |
| Output | Continuous Value | Probability (0โ1) |
| Activation Function | None | Sigmoid |
| Loss Function | Mean Squared Error | Binary Cross-Entropy |
| Decision Boundary | No | Yes |
๐ Advantages and Limitations
Advantages
Limitations
- Simple and computationally efficient.
- Easy to interpret.
- Produces probability estimates.
- Works well for linearly separable data.
- Supports regularization techniques.
- Cannot model highly nonlinear decision boundaries without feature engineering.
- Sensitive to multicollinearity.
- Performance decreases when classes overlap significantly.
- Requires sufficient training data for reliable probability estimation.
๐ Real-World Applications
| Application | Classification Task |
|---|---|
| ๐ง Spam Detection | Spam or Not Spam |
| ๐ฅ Medical Diagnosis | Disease Present or Absent |
| ๐ณ Fraud Detection | Fraudulent or Legitimate Transaction |
| ๐ Customer Churn Prediction | Leave or Stay |
| ๐ Sentiment Analysis | Positive or Negative Review |
| ๐ฆ Loan Approval | Approve or Reject |
๐ Logistic Regression Workflow
Collect Data
Preprocess Features
Train Logistic Regression Model
Estimate Probabilities
Apply Decision Threshold
Predict Class Labels
๐ป Practical Example
Logistic Regression Using Scikit-learn
from sklearn.linear_model import LogisticRegression
import numpy as np
# Sample training data
X = np.array([[1], [2], [3], [4], [5], [6]])
y = np.array([0, 0, 0, 1, 1, 1])
# Create model
model = LogisticRegression()
# Train model
model.fit(X, y)
# Predict class
prediction = model.predict([[3.5]])
# Predict probability
probability = model.predict_proba([[3.5]])
print("Predicted Class:", prediction[0])
print("Class Probabilities:", probability[0])โ ๏ธ Common Mistakes
- Using Logistic Regression for regression problems.
- Ignoring class imbalance.
- Not scaling numerical features when appropriate.
- Evaluating performance using accuracy alone on imbalanced datasets.
- Assuming a fixed threshold of 0.5 is always optimal.
Best Practice
Use feature scaling, cross-validation, and appropriate evaluation metrics such as Precision, Recall, F1-Score, and ROC-AUC when building Logistic Regression models. Adjust the classification threshold based on business requirements rather than relying solely on the default value.
๐ Summary
Summary
Logistic Regression is one of the most widely used classification algorithms because of its simplicity, interpretability, and efficiency. By transforming a linear combination of input features through the sigmoid function, it estimates class probabilities and predicts categorical outcomes. It performs exceptionally well for binary classification problems and often serves as a reliable baseline before applying more complex machine learning models.