Naïve Bayes Classifier in Machine Learning
1. Introduction to Naïve Bayes Classifier
The Naïve Bayes (NB) classifier is a probabilistic machine learning algorithm used for classification tasks. It is based on Bayes’ Theorem, which describes the probability of an event occurring given prior knowledge of related conditions. The algorithm is called “naïve” because it assumes that all features are independent of each other, which is rarely true in real-world applications. Despite this unrealistic assumption, Naïve Bayes performs well in various classification problems, especially in text classification, spam filtering, and sentiment analysis.
2. Why Use Naïve Bayes?
✔ Fast and efficient – Suitable for large datasets.
✔ Simple and easy to implement – Requires minimal computational resources.
✔ Works well with categorical & textual data – Commonly used in NLP and spam detection.
✔ Performs well even with small datasets – Can handle noisy data effectively.
✔ Requires less training data – Unlike deep learning, Naïve Bayes does not need massive data to perform well.
✔ Probabilistic Model – Provides probability estimates for predictions.
3. How Does Naïve Bayes Work?
🌟 The Core Idea: Bayes’ Theorem
The Naïve Bayes classifier is based on Bayes’ Theorem, which states: P(A∣B)=P(B∣A)×P(A)P(B)P(A | B) = \frac{P(B | A) \times P(A)}{P(B)}
Where:
- P(A | B) = Probability of event A occurring given that event B has occurred (Posterior Probability).
- P(B | A) = Probability of event B occurring given that event A has occurred (Likelihood).
- P(A) = Probability of event A occurring (Prior Probability).
- P(B) = Probability of event B occurring (Evidence).
📌 The goal of Naïve Bayes is to compute P(Class | Features), i.e., the probability of a data point belonging to a certain class, given its features. The class with the highest probability is chosen as the predicted class.
4. Types of Naïve Bayes Classifiers
1️⃣ Gaussian Naïve Bayes (GNB)
Used when the features follow a normal distribution (continuous data).
Example: Classifying iris flowers based on petal and sepal length.
📌 Formula for Likelihood (Gaussian Distribution): P(X∣C)=12πσ2e−(X−μ)22σ2P(X | C) = \frac{1}{\sqrt{2 \pi \sigma^2}} e^{-\frac{(X – \mu)^2}{2\sigma^2}}
Where:
- XX = feature value
- μ\mu = mean
- σ2\sigma^2 = variance
2️⃣ Multinomial Naïve Bayes (MNB)
Used for discrete data like word counts in text classification.
Example: Spam filtering, sentiment analysis.
📌 Formula for Likelihood (Multinomial Distribution): P(X∣C)=(N!)X1!X2!…Xn!P1X1P2X2…PnXnP(X | C) = \frac{(N!)}{X_1! X_2! … X_n!} P_1^{X_1} P_2^{X_2} … P_n^{X_n}
Where:
- XiX_i = Count of feature ii
- PiP_i = Probability of feature ii occurring in class CC
3️⃣ Bernoulli Naïve Bayes (BNB)
Used when features are binary (0 or 1).
Example: Email spam detection (presence or absence of words in an email).
📌 Formula for Likelihood (Bernoulli Distribution): P(X∣C)=Px(1−P)(1−x)P(X | C) = P^x (1 – P)^{(1 – x)}
Where:
- PP = Probability of feature occurring
- xx = Presence (1) or absence (0) of feature
5. Steps to Implement Naïve Bayes Algorithm
Step 1: Collect and Prepare Data
🔹 Data should be in tabular form (features + labels).
🔹 If using text, apply preprocessing (tokenization, stop-word removal).
Step 2: Convert Data into Probabilities
🔹 Compute prior probabilities for each class: P(Class)=Samples in ClassTotal SamplesP(Class) = \frac{\text{Samples in Class}}{\text{Total Samples}}
🔹 Compute likelihood using one of the Naïve Bayes formulas based on the data type (Gaussian, Multinomial, or Bernoulli).
Step 3: Apply Bayes’ Theorem to Calculate Posterior Probability
🔹 For each new sample, compute: P(Class∣Features)=P(Features∣Class)×P(Class)P(Features)P(Class | Features) = \frac{P(Features | Class) \times P(Class)}{P(Features)}
🔹 Assign the class with the highest probability to the new data point.
Step 4: Make Predictions
🔹 Compare the computed probabilities for all classes.
🔹 Assign the most probable class to the data point.
Step 5: Evaluate the Model
🔹 Use metrics like:
✅ Accuracy
✅ Precision, Recall, F1-score
✅ Confusion Matrix
✅ ROC Curve & AUC Score
6. Implementing Naïve Bayes in Python (Sklearn)
📌 Step 1: Import Libraries
import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import GaussianNB, MultinomialNB, BernoulliNB
from sklearn.metrics import accuracy_score, classification_report, confusion_matrix
📌 Step 2: Load Dataset
# Sample Dataset
data = {'Feature1': [2, 4, 6, 8, 10, 12, 14, 16],
'Feature2': [1, 3, 5, 7, 9, 11, 13, 15],
'Class': [0, 0, 0, 1, 1, 1, 1, 1]}
df = pd.DataFrame(data)
# Features & Target
X = df[['Feature1', 'Feature2']]
y = df['Class']
📌 Step 3: Split Data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
📌 Step 4: Train Naïve Bayes Model
# Using Gaussian Naïve Bayes
model = GaussianNB()
model.fit(X_train, y_train)
📌 Step 5: Make Predictions
y_pred = model.predict(X_test)
📌 Step 6: Evaluate Model
accuracy = accuracy_score(y_test, y_pred)
conf_matrix = confusion_matrix(y_test, y_pred)
report = classification_report(y_test, y_pred)
print(f'Accuracy: {accuracy:.2f}')
print('Confusion Matrix:')
print(conf_matrix)
print('Classification Report:')
print(report)
📌 MultinomialNB and BernoulliNB can be used for text-based datasets.
7. Advantages & Disadvantages
✅ Advantages
✔ Fast and efficient
✔ Works well for high-dimensional data (text classification, spam filtering)
✔ Performs well with small datasets
✔ Simple and easy to interpret
❌ Disadvantages
❌ Assumption of feature independence is rarely true
❌ Struggles with continuous and highly correlated features
❌ Not suitable for complex decision boundaries
8. Summary
✔ Naïve Bayes is a fast, efficient, and simple classification algorithm.
✔ It is based on Bayes’ Theorem and assumes feature independence.
✔ Works best for text classification, spam filtering, and sentiment analysis.
✔ Three types: Gaussian (GNB), Multinomial (MNB), and Bernoulli (BNB).
✔ Scikit-learn provides easy implementation. Mastering Naïve Bayes is essential for NLP and classification tasks!