Overfitting happens when a model learns the noise in its training data along with the signal. It scores brilliantly on data it has seen and poorly on data it has not. The classic sign is a wide gap between training and test scores. Beginners often respond by adding complexity, which usually makes the problem worse.
Ways to fight overfitting
- Collect more training data when possible, since memorisation is harder with more examples.
- Prefer simpler models first, and add complexity only when the simpler model clearly underfits.
- Use cross-validation to judge the model on several held-out folds instead of one lucky split.
- Apply regularisation, which penalises large coefficients and keeps the model from bending to every data point.
A model that is slightly worse in training but stable in testing is the better model. Judge by the test score, and be suspicious of perfection.