Python for Data Science

This deck is built to help you build a solid foundation in NumPy, the core library behind most data science work in Python. The cards walk you through array creation, indexing and slicing, reshaping, broadcasting, and universal functions — the everyday building blocks you'll reach for when working with numerical data. If you're moving from basic Python into data analysis, machine learning, or scientific computing, these are the concepts you'll need at your fingertips.

The questions are written in a short, recall-friendly format, so the deck works best as an active practice tool rather than a first read. Try to answer each prompt on your own before revealing the solution, and pay extra attention to the code-based cards, since recognizing what a function returns or how slicing behaves is faster learned by doing than by re-reading notes.

Because NumPy syntax is dense and easy to forget between sessions, spaced repetition will help far more than cramming. Aim for a short review every day or two in the early stages, then stretch the gaps as the cards start feeling familiar. Pair the deck with a Python notebook so you can quickly test any snippet you're unsure about — seeing the output once will usually lock in the answer for good.

NumPy is the foundational library for numerical computing in Python, and it revolves around the ndarray object. Arrays can be created from existing Python lists using np.array([1, 2, 3]), or built from scratch with convenience constructors such as np.zeros(5) for arrays of zeros, np.eye(3) for an identity matrix, and np.arange(0, 10, 2) for evenly stepped integer ranges. When you need a specific number of evenly spaced points over an interval, np.linspace(0, 1, 5) is the right tool, since it returns exactly the requested number of samples between the endpoints.

NumPy's indexing and slicing follow Python conventions but extend naturally to multiple dimensions. The expression arr[1, 2] selects the element in the second row and third column using zero-based indexing, while arr[:, 0] pulls out the entire first column as a 1D array. Slicing such as arr[1:4] follows the half-open rule, so the stop index is excluded. A defining feature of NumPy is broadcasting, which lets you combine arrays of different shapes without writing explicit loops. For instance, arr * 3 multiplies every element by the scalar 3, while arr > 5 produces a boolean array from an element-wise comparison.

Reshaping operations like arr.reshape(3, 4) rearrange elements into a new view whose total size must match the original, and arr.ravel() flattens any array back into 1D. Universal functions, or ufuncs, are the workhorses of element-wise computation: np.sqrt(arr) and np.exp(arr) apply the corresponding mathematical function to every entry. Conditional transformations can be expressed compactly with np.where(arr > 0, arr, 0), which keeps positive values and replaces the rest. NumPy also supports linear-algebra operations such as np.dot(a, b) (equivalently a @ b), the transpose via arr.T, vertical stacking with np.vstack, and random sampling through np.random.randint.

Pandas builds on NumPy to provide labeled, tabular data structures ideal for data analysis. The primary object is the DataFrame, which can be created directly from a dictionary such as pd.DataFrame({'col1': [1, 2], 'col2': [3, 4]}). Once you have a DataFrame, the two main ways of selecting data are df.loc[] and df.iloc[]. The loc accessor is label-based, using the actual index values and column names, whereas iloc is purely integer-position based. For example, df.loc[0:3, ['name', 'age']] returns rows 0 through 3 together with only the 'name' and 'age' columns, while df.iloc[:5, :2] selects the first five rows and first two columns by position.

Boolean indexing is the standard way to filter rows in Pandas. Writing df[df['age'] > 30] yields a DataFrame containing only the rows where the condition holds. Multiple conditions can be combined by wrapping each one in parentheses and joining them with & (and) or | (or), as in df[(df['age'] > 25) & (df['city'] == 'NYC')]. Sorting is straightforward with df.sort_values('price', ascending=False), and you can sort by multiple columns with different orders using df.sort_values(['col1', 'col2'], ascending=[True, False]).

DataFrame indices are flexible and can be modified as needed. df.set_index('col') promotes a column to the row index, df.reset_index(drop=True) restores the default integer index while discarding the old one, and df.rename(columns={'old_name': 'new_name'}) renames selected columns. You can also create new columns directly through vectorized arithmetic, such as df['total'] = df['price'] * df['quantity'], or by applying a custom function element-wise with df['col'].apply(lambda x: x ** 2). These operations together make Pandas the most natural tool for the interactive slicing and dicing of tabular data.

Real-world data is messy, and Pandas offers a rich toolkit for cleaning it. Duplicate rows can be flagged with df.duplicated(), which returns a boolean Series marking every row after the first occurrence as a duplicate, and removed with df.drop_duplicates(). When the same value appears in slightly different forms, the .str accessor provides vectorized string methods: df['col'].str.lower() converts entries to lowercase, and df['col'].str.strip() removes leading and trailing whitespace. Substitutions are handled by df['col'].replace({'old_val': 'new_val'}), which maps old values to new ones across the entire column.

Type conversion is another pillar of data preparation. df['col'].astype(int) casts a column to integers, while pd.to_datetime(df['date_col']) parses strings into proper datetime64 values. Once a column is datetime-typed, the .dt accessor exposes convenient components: df['date_col'].dt.year extracts the year, and df['date_col'].dt.dayofweek returns integers from 0 (Monday) to 6 (Sunday). For columns with a limited number of unique values, converting to the category dtype with df['col'].astype('category') reduces memory usage and can speed up groupby operations.

Missing data is a constant concern, and Pandas represents it as NaN, None, or NaT for datetimes. The function pd.isnull(value) checks whether a single value is missing, while df.isnull().sum() reports the count of NaNs per column and df.isna().any() indicates which columns contain at least one missing value. You can drop incomplete rows with df.dropna() or, more conservatively, df.dropna(how='all') when you only want to remove rows that are entirely empty. Imputation is equally flexible: df['col'].fillna(df['col'].mean()) substitutes the mean, fillna(method='ffill') performs forward-fill, and df['col'].interpolate(method='linear') fills gaps with linear interpolation between existing data points.

Matplotlib is the workhorse plotting library underlying most Python visualization. A basic line plot is created with plt.plot(x, y) followed by plt.show(). Before displaying, you can annotate the figure using plt.title('Title'), plt.xlabel('X'), and plt.ylabel('Y'). Other common plot types include plt.bar(categories, values) for vertical bar charts, plt.scatter(x, y) for individual data points, and plt.hist(data, bins=20) for histograms that show the distribution of a single variable across a chosen number of bins.

For more sophisticated statistical graphics, Seaborn builds on Matplotlib with higher-level functions. A correlation matrix can be visualized as a heatmap with sns.heatmap(df.corr(), annot=True, cmap='coolwarm'), where the color encoding and the optional numerical annotations make relationships between variables easy to read. sns.countplot(x='category', data=df) shows the number of observations within each category as bars, and sns.pairplot(df) generates a grid of scatter plots for every pair of numeric columns, with histograms along the diagonal for individual distributions.

Both libraries support layouts of multiple plots side by side. Calling fig, axes = plt.subplots(1, 2, figsize=(10, 4)) creates a figure with one row and two columns of axes that you can populate individually. Once a figure is ready, plt.savefig('plot.png', dpi=300, bbox_inches='tight') saves it to disk, with dpi controlling resolution and bbox_inches='tight' trimming excess whitespace. In Jupyter notebooks, running %matplotlib inline at the top of a notebook ensures that plots are rendered directly below the code cell, making exploration and storytelling both interactive and reproducible.

Group-based aggregation is central to data analysis, and Pandas implements the familiar split-apply-combine pattern through groupby. The expression df.groupby('category')['value'].mean() computes the mean of value for each unique category, and df.groupby('col').size() returns a Series with the number of rows in each group. Multiple summary statistics can be calculated at once with df.groupby('col').agg(['mean', 'sum', 'count']), which produces a DataFrame whose columns are the chosen aggregations. When you need a result that aligns with the original rows rather than a single row per group, transform is the right tool: df.groupby('group')['val'].transform(lambda x: (x - x.mean()) / x.std()) standardizes values within each group, useful for features that should be normalized relative to their own category. To run completely custom logic, df.groupby('col').apply(my_function) passes each group DataFrame to your function.

Combining data from multiple sources is another core capability. The function pd.merge(df1, df2, on='key') performs an inner join by default, keeping only rows whose key appears in both DataFrames. The how argument controls the join style: 'left' keeps all rows from the first DataFrame, 'right' keeps all rows from the second, and 'outer' keeps all rows from both, filling missing matches with NaN. When the key columns have different names, you can specify them separately with left_on and right_on, as in pd.merge(df1, df2, left_on='id', right_on='user_id'). For index-based joins, df1.join(df2, how='inner') aligns the two DataFrames on their index.

Stacking and reshaping cover cases where data needs to be reorganized rather than joined by key. pd.concat([df1, df2]) stacks DataFrames vertically by default, appending rows, while pd.concat([df1, df2], axis=1) joins them horizontally, aligning on the index. Pivot tables summarize data along two dimensions: df.pivot_table(values='sales', index='region', columns='quarter', aggfunc='sum') produces a matrix of total sales broken down by region and quarter. The inverse operation, df.melt(id_vars=['id'], value_vars=['col1', 'col2']), unpivots columns into rows, converting wide-format data into long format suitable for many plotting libraries and tidy-data workflows.

Before feeding data to machine-learning models, raw columns usually need to be transformed into more informative features. The simplest step is creating new columns through vectorized arithmetic, such as df['total'] = df['price'] * df['quantity'], or through the element-wise application of a custom function with df['col'].apply(lambda x: x ** 2). These operations are fast because they run on underlying NumPy arrays rather than row-by-row Python loops.

Categorical variables typically need to be converted into numerical form. One-hot encoding creates a separate binary column for each category, and in Pandas this is done with pd.get_dummies(df, columns=['col']). Continuous variables, on the other hand, are often discretized into bins. pd.cut(df['age'], bins=[0, 18, 35, 60, 100], labels=['child', 'young', 'middle', 'senior']) assigns each value to a predefined interval, while pd.qcut(df['value'], q=4, labels=['Q1', 'Q2', 'Q3', 'Q4']) creates equal-frequency bins such as quartiles, so each bin contains roughly the same number of observations.

Feature scaling is essential for algorithms that are sensitive to the magnitude of features, such as K-nearest neighbors or support vector machines. The two most common approaches are min-max scaling, which rescales values to the range \([0, 1]\) using \(x' = \frac{x - x_{\min}}{x_{\max} - x_{\min}}\), and standardization, which centers values around zero with unit variance using \(z = \frac{x - \mu}{\sigma}\). Both are easily applied through scikit-learn: instantiate a MinMaxScaler or StandardScaler, call fit_transform on a column reshaped as df[['col']], and assign the result back. Choosing the right scaler depends on the model: min-max is common for neural networks and image data, while standardization is generally preferred for models that assume approximately Gaussian inputs.

Before any serious analysis, you need a quick overview of a DataFrame. df.shape returns a tuple of (rows, columns), df.head(10) shows the first ten rows, and df.dtypes lists the data type of each column. The methods df.info() and df.describe() are particularly useful: info() prints column names, non-null counts, dtypes, and memory usage, giving a one-glance health check of the data, while describe() returns summary statistics such as count, mean, standard deviation, and the 25%, 50%, and 75% percentiles for every numeric column.

For categorical or discrete columns, df['col'].value_counts() returns the frequency of each unique value sorted in descending order, while df['col'].unique() lists the distinct values and df['col'].nunique() returns their count. The correlation matrix df.corr() produces pairwise Pearson correlation coefficients for all numeric columns and is often the first step toward understanding linear relationships between features.

Jupyter notebooks provide a rich environment for iterative data science, and a few built-in tools make them even more productive. The line magic %timeit runs a statement many times and reports its average execution time, which is ideal for benchmarking small snippets, while the cell magic %%time measures the wall time and CPU time of an entire cell in a single run. Keyboard shortcuts speed up navigation: Shift + Enter runs the current cell and advances, pressing Esc enters command mode, and A inserts a new cell above the active one. Finally, data flows in and out of notebooks through simple I/O functions: pd.read_csv('file.csv') loads a CSV (or a TSV with sep='\t'), and df.to_csv('output.csv', index=False) writes a DataFrame to disk, with index=False preventing the index from being saved as an extra column.