Overfitting Explained

The student who memorised the textbook

Imagine two students preparing for a maths exam. The first student understands the methods: she can solve any problem of the type she studied, even ones she has never seen. The second student took a different approach. He memorised the worked answers to every single practice problem, word for word. On the practice problems, he scores 100%. But in the real exam, with new numbers, he falls apart, because he never actually learned how to solve anything. He only learned those specific answers.

That second student is doing exactly what an overfit machine-learning model does. Overfitting is one of the most important and most misunderstood ideas in all of AI, and understanding it deeply separates people who can build trustworthy models from people who are fooled by impressive-looking demos.

This lesson assumes you already know the basics of how a model trains and is tested. If you want a refresher first, What Is Machine Learning? lays the groundwork.

What overfitting actually is

When a model trains, its goal is to learn the real pattern connecting inputs to outputs, the genuine signal that will hold true for new data. But real-world training data also contains noise: random quirks, measurement errors, and coincidences that mean nothing. For example, in a small dataset of house prices, it might just so happen that the three blue houses all sold for a high price. That is a coincidence, not a rule about the colour blue.

Overfitting happens when a model learns the noise along with the signal. It starts treating coincidences as if they were real rules. The blue-house coincidence gets baked into the model. On the training data, this looks great, because the model can now "explain" every example perfectly, including the flukes. But the moment you show it new data, those memorised flukes do not apply, and performance collapses.

In short: an overfit model has learned its training data too well. It fits the specific examples rather than the underlying truth. The thing we actually want is called generalisation, the ability to perform well on data the model has never seen. Overfitting is the enemy of generalisation.

How to spot it: the train-test gap

You cannot judge overfitting by looking at training performance alone, because an overfit model looks fantastic on its training data. The trick is to compare two numbers:

• Training accuracy: how well the model does on the data it learned from.
• Test accuracy: how well it does on fresh, held-out data it never saw during training.

Here is the tell-tale signature:

Situation	Training score	Test score	Diagnosis
Good fit	High	High	The model generalises well
Overfitting	Very high	Much lower	Memorised the training data
Underfitting	Low	Low	Too simple to learn the pattern

A big gap between high training accuracy and low test accuracy is the classic fingerprint of overfitting. If a model scores 99% on training data but only 70% on test data, that 29-point gap is screaming that the model has memorised rather than learned. This is also why you must always keep a separate test set the model never sees during training; testing on training data hides overfitting entirely.

The opposite problem: underfitting

It would be easy to conclude "make the model simpler to avoid overfitting" and stop there, but that overcorrects into the opposite mistake. Underfitting is when a model is too simple to capture the real pattern at all. It does poorly on both the training data and the test data, because it never learned enough in the first place.

Picture trying to fit a perfectly straight line through data that actually curves. No matter how you position the straight line, it cannot follow the curve, so it is wrong everywhere. That is underfitting: not enough flexibility.

So model-building is a balancing act between two failures:

• Too simple → underfitting → misses the pattern.
• Too complex → overfitting → memorises the noise.

The sweet spot in the middle is a model just flexible enough to capture the true pattern but not so flexible that it starts memorising flukes. Experts call this tension the bias-variance tradeoff. A model that is too rigid has high "bias" (it makes systematic errors); a model that is too flexible has high "variance" (its answers swing wildly with small changes in the data). Good models balance the two.

How engineers fight overfitting

Because overfitting is so common, a whole toolkit exists to push back against it. No single tool is magic; engineers usually combine several.

1. Get more and more varied data. This is often the most powerful fix. The more examples a model sees, the harder it is to memorise quirks, because the quirks get drowned out by the genuine pattern repeating across many cases. A model can memorise 50 houses; it struggles to memorise 5 million. Where collecting new data is impossible, engineers sometimes use data augmentation, creating slightly altered copies of existing examples (a photo flipped or rotated) to enlarge the dataset. The variety of the data matters as much as the amount, which connects to the issues raised in Training Data and Bias in AI.

2. Simplify the model. A model with fewer parameters has less room to memorise. For a decision tree, this means limiting how deep it can grow. For other models, it means choosing a less complex shape. The goal is to give the model just enough capacity to learn the real pattern and no more.

3. Use regularisation. This is a clever mathematical penalty added during training that discourages the model from relying too heavily on any single feature or from using extreme settings. In effect, it nudges the model towards simpler, smoother solutions that are less likely to chase noise. In neural networks, a popular related trick called dropout randomly switches off parts of the network during training, forcing it to learn robust patterns rather than fragile memorised ones. You can read more about these systems in Neural Networks Explained.

4. Stop training at the right time. A model often keeps improving on training data long after it has stopped improving on validation data. Watching a validation set (held-out data checked during development) lets engineers stop training at the point where test performance is best, before overfitting sets in. This is called early stopping.

5. Use cross-validation. Instead of trusting a single train-test split, engineers rotate which slice of data is held out, training and testing several times. Averaging the results gives a more honest picture of how the model truly generalises.

Why this matters in the real world

Overfitting is not just a classroom curiosity; it causes real failures. A medical model that overfits its training hospital's data may work brilliantly there and then fail at a different hospital with slightly different patients or equipment. A trading model that overfits past market data may look like a money machine in testing and lose money the moment markets behave even slightly differently. In every case, the danger is the same: the model looked amazing on the data it knew and fell apart on the data that actually mattered.

The honest takeaway is this. Whenever you see a claim that some AI achieved near-perfect performance, your first question should be: perfect on what data? If the answer is "the data it trained on," that is not impressive at all; it might just be overfitting wearing a disguise. Real intelligence, in machines as in people, shows itself not by repeating memorised answers but by handling something new.