A model that has seen the answers during training will score well on those answers. That score says nothing about new data. The train-test split fixes this by holding out part of the data before training, then testing only on the held-out part. It is a simple habit, and skipping it is the most common beginner mistake in machine learning.
Doing the split properly
- Split before any preprocessing that learns from data, such as scaling or imputation, or information leaks into the test set.
- Keep the test set untouched during model selection. Repeated peeking turns it into training data.
- Stratify the split for classification, so each class appears in both parts in roughly its original proportion.
- Report the test score alongside the training score. A large gap between them signals overfitting.
When a model scores far better in training than in testing, it has memorised, not learned. Trust the test score, and fix the model, not the split.