The quality and form of input features often matter more than algorithm choice. Feature engineering creates, transforms, or selects features to expose useful signal to the model, including extracting date and time components, binning continuous variables, creating interaction features, and encoding categorical variables. One-hot encoding turns each category into a binary column with a 1 indicating the presence of that category, while label encoding assigns integers; scikit-learn provides OneHotEncoder and pandas offers pd.get_dummies for the same purpose. Good feature engineering often has a bigger impact than algorithm choice.
Numerical features often need scaling so that no single feature dominates. Normalization (min-max scaling) rescales features to a fixed range, typically [0, 1] using \( (x - \min) / (\max - \min) \), while standardization (Z-score) centers features at zero mean with unit variance using \( (x - \mu) / \sigma \). Scaling is essential for distance-based algorithms like KNN and SVM, and for gradient-based methods like neural networks and logistic regression, where features with larger magnitudes can dominate learning.
Real datasets have imperfections that must be addressed. Missing data can be removed, imputed with mean, median, or mode, or filled using methods like KNN imputation, with scikit-learn's SimpleImputer providing convenient strategies; indicator variables can flag missingness as an additional feature. Class imbalance, where one class dominates, biases models toward the majority class and can be mitigated by resampling such as oversampling the minority with SMOTE, undersampling the majority, class weighting in the loss, or threshold tuning. Outliers—points that significantly deviate—may be data errors or genuine rare events; robust methods such as Huber loss, RANSAC, or Isolation Forest are less sensitive to them. Data augmentation artificially expands training data via label-preserving transformations such as rotations and flips for images, or synonym replacement and back-translation for text.
Feature selection methods reduce dimensionality, noise, and overfitting. Filter approaches use correlation, chi-square, or mutual information to rank features, with SelectKBest selecting the top-k by mutual information. Wrapper methods like recursive feature elimination (RFE) recursively remove the least important features by retraining a model. Embedded methods such as Lasso or tree importances perform selection as part of training. Scikit-learn's Pipeline object chains transformers and an estimator so the same preprocessing is consistently applied during training and prediction, preventing data leakage. Data leakage, where information from outside the training set bleeds into the model, produces falsely good evaluation scores and is a common cause of deployment failures. The curse of dimensionality describes how data becomes sparse and distances lose meaning in very high dimensions, motivating dimensionality reduction via PCA or UMAP. In production, data drift occurs when input distributions change and concept drift occurs when the relationship between features and target changes, both requiring monitoring and retraining.