Real-world data is messy, and Pandas offers a rich toolkit for cleaning it. Duplicate rows can be flagged with df.duplicated(), which returns a boolean Series marking every row after the first occurrence as a duplicate, and removed with df.drop_duplicates(). When the same value appears in slightly different forms, the .str accessor provides vectorized string methods: df['col'].str.lower() converts entries to lowercase, and df['col'].str.strip() removes leading and trailing whitespace. Substitutions are handled by df['col'].replace({'old_val': 'new_val'}), which maps old values to new ones across the entire column.
Type conversion is another pillar of data preparation. df['col'].astype(int) casts a column to integers, while pd.to_datetime(df['date_col']) parses strings into proper datetime64 values. Once a column is datetime-typed, the .dt accessor exposes convenient components: df['date_col'].dt.year extracts the year, and df['date_col'].dt.dayofweek returns integers from 0 (Monday) to 6 (Sunday). For columns with a limited number of unique values, converting to the category dtype with df['col'].astype('category') reduces memory usage and can speed up groupby operations.
Missing data is a constant concern, and Pandas represents it as NaN, None, or NaT for datetimes. The function pd.isnull(value) checks whether a single value is missing, while df.isnull().sum() reports the count of NaNs per column and df.isna().any() indicates which columns contain at least one missing value. You can drop incomplete rows with df.dropna() or, more conservatively, df.dropna(how='all') when you only want to remove rows that are entirely empty. Imputation is equally flexible: df['col'].fillna(df['col'].mean()) substitutes the mean, fillna(method='ffill') performs forward-fill, and df['col'].interpolate(method='linear') fills gaps with linear interpolation between existing data points.