Machine Learning Python NLP Classification
Spam Email Detection.
A machine learning project focused on identifying spam emails while minimizing the risk of legitimate messages being incorrectly classified as spam.
Data Science Bootcamp Project · Python · Scikit-learn
01. The problem
Spam detection is a common classification problem, but not all classification errors have the same consequences.
In this project, the main objective was not simply to maximize overall accuracy. I wanted to build a model capable of detecting unwanted emails while ensuring that fewer than 1% of legitimate emails were incorrectly classified as spam.
This requirement makes false positives particularly important: an incorrectly classified legitimate email could end up in the spam folder, causing the user to miss an important message.
02. The dataset
The project uses a labelled dataset of email messages divided into two categories:
- Legitimate (0): Regular email messages.
- Spam (1): Unwanted or unsolicited emails.
I started by exploring the dataset and examining the distribution of the target variable. This initial analysis helped identify the class imbalance and provided context for the modelling and evaluation stages.
03. Methodology
The project followed a series of steps, starting with text exploration and ending with an independently evaluated classification pipeline.
Text preprocessing
Email messages contain a considerable amount of textual noise. To prepare the data, I implemented a preprocessing function including:
- Converting text to lowercase.
- Replacing URLs and email addresses with standard tokens.
- Removing punctuation, numbers and special characters.
- Removing English stopwords.
- Applying lemmatization using NLTK.
- Removing the "subject" prefix when it appears at the beginning of a message.
Feature extraction
I used CountVectorizer to convert email text into numerical features using a Bag-of-Words representation. The vocabulary was limited to 1,000 features to keep the model relatively lightweight.
Model training
The selected classifier was a Random Forest Classifier from Scikit-learn. I also experimented with different vocabulary sizes and class weights to explore their impact on performance.
The final implementation uses a Scikit-learn Pipeline to combine text preprocessing, vectorization and classification into a single workflow.
Threshold selection and evaluation
Instead of relying on the default probability threshold of 0.5, I evaluated different thresholds to find a better balance between false positives and spam detection.
I used separate training, threshold-selection and final evaluation subsets. The threshold was selected using the development data, while the final evaluation was performed on a separate holdout set.
The main evaluation metrics were the False Positive Rate (FPR), Spam Recall, Accuracy, Precision and F1-score.
04. Results
The final model was evaluated on an independent holdout set containing 1,035 emails: 735 legitimate messages and 300 spam messages.
A probability threshold of 0.65 achieved the following results:
- False Positive Rate: 0.82%
- Spam Recall: 80.00%
- Accuracy: 93.62%
- Decision threshold: 0.65
The model achieved the main project objective on the independent evaluation set, keeping the False Positive Rate below 1%.
This came with a trade-off: the model detected 80% of spam messages, meaning that some unwanted emails were still classified as legitimate.
The experiment highlights why accuracy alone is not enough when evaluating a classification model. The choice of evaluation metrics and decision threshold should reflect the actual cost of different types of errors.
05. What I learned
This project helped me understand several important aspects of practical machine learning, particularly when working with text data.
- How to transform unstructured text into numerical features using Bag of Words.
- How preprocessing techniques can help prepare text for machine learning algorithms.
- Why a confusion matrix and class-specific metrics are important when evaluating classification models.
- How changing the decision threshold affects the balance between false positives and recall.
- How Scikit-learn Pipelines can combine preprocessing, feature extraction and model training into a reusable workflow.
Future improvements
There are several ways this project could be extended:
- Compare Random Forest with other models.
- Experiment with TF-IDF instead of Bag of Words.
- Perform more systematic hyperparameter tuning.
- Evaluate the stability of the False Positive Rate across different datasets.
- Build a simple interactive email classification application using Streamlit.
06. Explore the code
The repository contains the Jupyter Notebook with the exploratory analysis, text preprocessing, model training and evaluation process.
You can explore the code and review the implementation on GitHub.