Real datasets arrive messy: missing values, inconsistent spellings, duplicate rows, and dates in three formats. A model trained on that data learns the mess as if it were signal. Cleaning with pandas means finding these problems systematically before any modelling starts, and keeping a record of every change so the work can be repeated.
A cleaning pass in order
- Profile first: row counts, missing values per column, and duplicate rows, so you know the scale of the mess.
- Standardise text: one casing, one spelling for each category, and trimmed whitespace, before any grouping.
- Handle missing values deliberately: drop, fill, or flag, depending on why the value is missing, and note the choice.
- Parse dates into one format and check ranges, since a birth year of 2090 is a data entry error, not an outlier.
Clean in a script, not by hand-editing the file. Hand edits cannot be repeated, and unrepeatable cleaning cannot be trusted.