Python is the dominant language in data science, supported by a layered ecosystem. NumPy provides N-dimensional arrays and fast vectorized operations; pandas offers DataFrames for tabular manipulation; scikit-learn implements classical machine learning algorithms with a consistent API; matplotlib is the foundational plotting library, complemented by seaborn for statistical graphics and Plotly for interactive visualizations; and Jupyter Notebooks provide an interactive environment that interleaves code, results, and narrative. R remains popular in academia and finance for statistical modeling, while SQL is the lingua franca for querying and manipulating relational databases, supporting joins to combine rows from related tables, GROUP BY for aggregation, and window functions for calculations across related rows.
Behind these familiar interfaces lies substantial infrastructure. ETL — Extract, Transform, Load — moves and shapes data into analytical systems; ELT flips the order, doing transformation in the warehouse where compute is cheap and the raw data is preserved. Storage architectures include data warehouses, central repositories optimized for analytical reads; data lakes, which store raw structured and unstructured data at scale; and lakehouses, a hybrid exemplified by Delta Lake and Apache Iceberg. Distributed processing is enabled by Apache Hadoop, with HDFS providing fault-tolerant storage and MapReduce offering a parallel programming model; Apache Spark supersedes MapReduce with in-memory computation; Apache Kafka handles real-time streaming; and Apache Airflow orchestrates workflows. Cloud platforms reduce the operational burden: Snowflake separates storage and compute in a managed warehouse, BigQuery offers Google’s serverless alternative, and Databricks unifies analytics on Spark. dbt applies SQL and Jinja templating for in-warehouse transformations, and feature stores centralize curated, versioned features so training and inference see the same definitions. Hadoop's broader ecosystem, including HDFS and MapReduce, remains relevant in legacy and on-premise installations.
Model parameters are learned from data during training, whereas hyperparameters such as the learning rate or tree depth are configuration choices set before training. Tuning them effectively is its own discipline: grid search exhaustively tries combinations, random search samples them, and Bayesian optimization builds a probabilistic model of performance to choose promising points efficiently. Once trained, models enter the MLOps lifecycle: deployment, monitoring, and retraining. Online inference serves predictions in real time, often behind a REST API, while batch inference generates predictions on bulk data at scheduled intervals. Serving systems such as TensorFlow Serving and NVIDIA's Triton Inference Server optimize throughput and latency, and ONNX provides a portable model interchange format so a model trained in one framework can run in another. A reusable feature pipeline computes features identically for training and inference, preventing skew. Data versioning with tools like DVC and lifecycle management with MLflow keep experiments, datasets, and models reproducible.
Models degrade as the world changes. Model drift is a decline in performance over time, driven by data drift (changes in input distributions), concept drift (changes in the relationship between inputs and outputs), or both. Monitoring catches both the symptoms (accuracy drops) and the causes (distribution shifts in features). Data quality monitoring platforms such as Great Expectations continuously validate data against expectations, complementing observability tooling that infers a system’s internal state from its outputs. Impact is measured through A/B testing, which compares two treatments to estimate causal effects, complementing offline metrics with real-world evidence.