How Spam Filters Work

The mail that nobody asked for

Every day, billions of junk emails are sent: fake prizes, dodgy links, scams and adverts. Yet your inbox usually stays fairly clean. The reason is a quiet piece of AI working behind the scenes called a spam filter.

A spam filter has one job. For every message that arrives, it must decide: is this spam (junk) or ham (a real, wanted message)? A program that sorts things into groups like this is called a classifier. Spam filtering is one of the oldest and most successful uses of machine learning. If you are new to the idea of computers learning from examples, start with What Is Machine Learning?.

The first idea: hand-written rules

The earliest filters used rules that humans wrote by hand. For example:

• If the subject line shouts in ALL CAPITALS, add some "spam points".
• If the message contains words like "free money" or "winner", add more points.
• If there are lots of links to strange websites, add more.

If a message collected enough points, it was sent to the junk folder.

Rules are simple and easy to understand, but they have a big weakness. Spammers quickly learn the rules and dodge them. They write "fr€€" instead of "free", or "w1nner" instead of "winner". The humans then have to write new rules, and the spammers change again. This is an endless game of cat and mouse, and pure rules slowly lose it.

The better idea: learning from examples

Modern filters mostly learn instead of following fixed rules. The process looks like this:

1. Collect a huge set of emails that are already labelled spam or not spam.
2. Let the program study which features show up more often in each group.
3. Use what it learned to score brand-new emails it has never seen.

This is exactly the "learn from labelled examples" approach explained in Teaching Machines with Examples. Because the filter learns from data, it can pick up subtle clues a human might never think to write down as a rule.

Thinking in probabilities

A classic and still-useful method is based on probability. The idea is to ask: given the words in this email, how likely is it to be spam?

Imagine the filter has read a million labelled emails. It notices that the word "lottery" appears in 8 out of 10 spam messages but almost never in real ones. So "lottery" becomes a strong spam clue. The word "meeting", on the other hand, shows up far more in real mail, so it pulls the score the other way.

For a new email, the filter looks at every word and adds up the evidence:

• Words that are common in spam push the score towards spam.
• Words that are common in real mail push the score towards not spam.

If the total crosses a chosen line, the message goes to the junk folder. This famous technique is called a naive Bayes classifier. It is "naive" because it pretends each word is independent of the others, which is not really true, but the trick works surprisingly well in practice.

More than just words

Real filters look at far more than the text. Useful features include:

• The sender's address and reputation (does this server send lots of spam?)
• Whether the links lead to known dangerous sites
• Hidden parts of the email called headers, which can reveal a faked sender
• The ratio of images to text, and odd formatting
• Whether thousands of people have already reported this exact message

No single feature is enough on its own. Plenty of honest emails contain the word "free". The power comes from combining many weak clues into one confident decision, the same principle behind How Recommendation Systems Work.

Two kinds of mistake

A filter can be wrong in two different ways, and the difference matters a lot.

• A false negative: a spam message sneaks into your inbox. Annoying, but mostly harmless.
• A false positive: a real message, maybe from a friend, a teacher or a doctor, gets wrongly thrown into the spam folder. You might never see it.

False positives are the dangerous mistake. Because of this, filters are deliberately tuned to be cautious about blocking mail. They would rather let a little spam through than hide something you actually needed. That trade-off is why you should occasionally check your spam folder.

You are part of the system

Every time you press "report spam" or "not spam", you are doing something powerful: you are giving the filter a fresh, correctly labelled example. Multiply that by millions of users, and the filter is constantly retrained on the newest spammer tricks.

This is a feedback loop. The spammers adapt, the filter learns from new reports, and the cycle continues. Spam filtering will never be "finished", but thanks to machine learning it stays good enough that most junk never reaches your eyes.

The honest limits

Spam filters are clever, not magic. They cannot truly understand meaning, so a carefully worded scam that uses normal-looking language can slip through. They can be biased by their training data, occasionally flagging unusual but genuine mail. And because they react to patterns, a new style of attack can work for a while before filters catch up.

That is why the smartest defence is a partnership: a good filter doing the heavy lifting, plus a careful human who never clicks a suspicious link, even when a message lands safely in the inbox.