Pandas DataFrame Operations

This deck introduces you to one of the most widely used tools in Python data analysis: the Pandas DataFrame. The cards walk you through the fundamentals, starting with what a DataFrame is and which library provides it, then moving into the practical skills you'll use every day. You'll learn how to build DataFrames from dictionaries, lists, and external files like CSV, Excel, and JSON, as well as how to save your results back out to disk.

Once you're comfortable creating DataFrames, the deck shifts to exploring and inspecting them. Cards cover handy methods and attributes for peeking at your data, checking its shape, listing columns and index labels, viewing data types, and pulling up a quick summary. These are the everyday commands you'll reach for whenever you load a new dataset and want to understand what you're working with.

This deck is a great fit if you're new to Pandas or transitioning from another data tool and want to build a solid foundation in DataFrame operations. To get the most out of it, try writing out the code for each question by hand before flipping the card, even if it's just on a scrap of paper. Pairing the flashcards with a short coding session between study rounds will help the syntax stick much faster than reading alone.

The Pandas library (imported as import pandas as pd) provides the DataFrame, a two-dimensional, size-mutable, tabular data structure with labeled rows and columns. Think of it as a spreadsheet or a SQL table that lives in memory: columns can hold different dtypes, and both rows and columns are indexed, which makes label-aware selection natural. The DataFrame sits at the center of nearly every data-analysis workflow in Python.

You can build a DataFrame from many sources. The simplest in-memory options pass a dictionary of lists — pd.DataFrame({'col1': [1, 2], 'col2': [3, 4]}) — where each key becomes a column label. You can also build one from a list of dictionaries, in which case missing keys across rows are filled with NaN. For data on disk, Pandas exposes readers for the common formats: pd.read_csv('file.csv') for comma-separated files, pd.read_excel('file.xlsx', sheet_name='Sheet1') for spreadsheets, and pd.read_json('file.json') for JSON. For line-delimited JSON you may need to pass orient='records'. Writing back is just as direct: df.to_csv('file.csv', index=False) saves a frame to disk while skipping the row index column.

Before manipulating a DataFrame you usually inspect it. df.head() and df.tail(n) peek at the first and last rows (default five). df.shape returns a tuple (n_rows, n_cols), df.columns lists the column labels, df.index exposes the row index, and df.dtypes reports the dtype of every column. For a single, human-readable summary, df.info() prints the shape, dtypes, non-null counts, and memory footprint in one shot.

Selecting data comes in two flavors: column access and row access. A single column is retrieved with df['col'] or the attribute form df.col, both of which return a one-dimensional Series. Multiple columns are requested by passing a list — df[['col1', 'col2']] — and produce a DataFrame. For rows, df.loc[row_labels, col_labels] does label-based selection while df.iloc[row_positions, col_positions] does integer-position-based selection; iloc[0] returns the first row, and iloc[5:10] slices rows five through nine (the end is exclusive). For fast scalar access, df.at['row', 'col'] and df.iat[2, 3] are quicker than loc and iloc respectively.

Filtering uses boolean masks. Equality with a value gives df[df['col'] == value], multiple AND-conditions are combined with &, and OR-conditions use |; each mask must be parenthesized so Python parses them correctly. Membership in a list is expressed with df[df['col'].isin(['a', 'b', 'c'])], missing values with df['col'].isna() (or .notna() for the inverse), and pattern matches with df['col'].str.contains('pattern', regex=True). As a shortcut, df.query('a > 0 and b < 10') lets you write the filter as a string expression that references column names directly without the df. prefix. When you need to assign to a filtered subset, prefer df.loc[mask, 'col'] = new_value over chained assignment like df[mask]['col'] = new_value, which can trigger a SettingWithCopyWarning and may silently fail.

Adding a column is as simple as assigning to a new key: df['new'] = df['a'] + df['b']. To remove one, use df.drop('col', axis=1) or the in-place shortcut del df['col']. Renaming is handled by df.rename(columns={'old': 'new'}, inplace=True), which also accepts a function for bulk renames — df.rename(columns=str.lower) lowercases every column. When the dtype of a column needs to change, df['col'] = df['col'].astype(int) converts it; common targets include int, float, str, the memory-efficient 'category', and the time-aware datetime64[ns].

Many DataFrame methods return a new frame rather than mutating the existing one. The parameter inplace=True tells the method to modify df directly — for example, df.drop('col', axis=1, inplace=True). Without it, you rebind the name: df = df.drop('col', axis=1). Both produce the same observable result, but only one keeps the original object identity. Knowing this distinction avoids subtle bugs, especially when chained indexing could trigger a SettingWithCopyWarning.

Sorting and reindexing control the order of rows. df.sort_values('col') orders by a column, with ascending=False flipping the direction; passing a list such as df.sort_values(['a', 'b'], ascending=[True, False]) sorts by multiple keys with independent directions. df.sort_index() instead sorts by the row index. To discard the current index and replace it with the default integer range, use df.reset_index(drop=True); the drop=True flag throws the old index away. Conversely, df.set_index('col') promotes an existing column to the index, which is especially useful for time-series work where the index becomes a DatetimeIndex.

The quickest overview of a numeric frame is df.describe(), which returns count, mean, standard deviation, min, quartiles, and max for every numeric column. For individual reductions, df.mean() computes the mean of each column, df.count() returns non-null counts per column, and df['col'].nunique() counts distinct values in a column. The distinct values themselves can also be inspected with df['col'].unique(), which returns a NumPy array, or with df['col'].value_counts(), which produces a frequency Series sorted in descending order by default.

Aggregation is governed by the axis argument. With the default axis=0, a function operates down the rows and produces one value per column; with axis=1, it operates across the columns and produces one value per row. df.apply(np.sum) therefore sums each column, while df.apply(np.sum, axis=1) sums each row. For element-wise work, df.applymap(func) (renamed df.map() in Pandas 2.1 and later) hits every cell, and on a Series both df['col'].map(func) and df['col'].apply(func) do the same per element, with the extra ability for map to accept a dict or Series for value substitution. To rewrite values globally, df.replace({'old1': 'new1', 'old2': 'new2'}) maps old values to new ones.

Several helpers complete the analysis toolbox. df['col'].rank(method='average') assigns ranks, with average, min, max, first, and dense as the available tie-breaking methods. df.corr(numeric_only=True) and df.cov(numeric_only=True) return the pairwise correlation and covariance matrices of the numeric columns. Standardizing a column to a z-score is a one-liner: (df['v'] - df['v'].mean()) / df['v'].std(). For sampling, df.sample(n=5) draws five random rows and df.sample(frac=0.1, random_state=42) takes 10%; passing random_state makes the draw reproducible. To clip values to a sensible range, df['v'].clip(lower=0, upper=100) caps them without dropping them. Iterating over rows is possible with for index, row in df.iterrows(): ..., which is convenient but slow; for row in df.itertuples(): ... is the faster namedtuple alternative, and vectorized operations should be preferred whenever feasible.

Real datasets are rarely clean. df.isna() produces a boolean DataFrame marking every NaN cell, and df.isna().sum() counts NaNs per column so you can see where the holes are. To detect rows that are exact duplicates of earlier rows, df.duplicated() returns a boolean Series, and df.duplicated().sum() totals them. Removing duplicates is a one-liner: df.drop_duplicates() drops all repeat rows, while df.drop_duplicates(subset=['col1', 'col2']) considers only the listed columns when deciding what counts as a duplicate.

For handling missing values, two complementary strategies exist: drop them or fill them. df.dropna() removes any row containing a NaN, and passing how='all' restricts the drop to rows that are entirely NaN. df.dropna(axis=1) instead drops columns with missing values. Filling is done with df.fillna(0) for a global constant, or with a per-column dict like df.fillna({'a': 0, 'b': -1}) for column-specific values. For ordered data such as time series, forward-fill propagates the last valid observation with df.ffill() (or the legacy df.fillna(method='ffill')), while backward-fill carries the next valid value backward using df.bfill().

Infinite values produced by division-by-zero or numerical overflow should be treated as a kind of missing data. df.replace([np.inf, -np.inf], np.nan) converts them to NaN so they can be handled by the same dropna/fillna pipeline. Equality between two frames can be checked with df1.equals(df2), which returns a single boolean, or element by element with df1 == df2, which returns a boolean DataFrame. Whenever you need an independent copy, df.copy() creates a deep copy; assigning slices without copying can produce a SettingWithCopyWarning when a later write may or may not affect the original frame.

The Split-Apply-Combine paradigm is encoded in df.groupby('key'). After grouping, aggregations like df.groupby('key').mean(numeric_only=True) or .sum() collapse each group into a single row; grouping by multiple columns, df.groupby(['k1', 'k2']).sum(), produces a MultiIndex on the result. To iterate over the groups, write for name, group in df.groupby('key'): .... When several aggregations are needed per group, df.groupby('key').agg({'a': 'mean', 'b': ['min', 'max']}) returns all of them in one call, and named aggregation df.groupby('key').agg(mean_a=('a', 'mean'), max_b=('b', 'max')) assigns clean column names to each result.

Two advanced groupby methods preserve the shape of the original frame. df.groupby('key')['v'].transform('mean') broadcasts the per-group mean back to every row, aligning the result with the input index — useful for adding group-normalized columns. df.groupby('key').filter(lambda g: g['v'].mean() > 0) instead drops entire groups that fail a boolean condition. Combining frames is done either by stacking or joining. pd.concat([df1, df2]) stacks rows vertically, with ignore_index=True renumbering rows from zero; passing axis=1 instead glues frames side-by-side as new columns. For relational joins, pd.merge(df1, df2, on='key') works like a SQL join; the how parameter takes 'left', 'right', 'inner' (the default), or 'outer'. When the join keys have different names, use left_on='id1', right_on='id2'. df1.join(df2) is a convenience that defaults to merging on the index, while merge is the more general form.

Reshaping rotates the layout of a frame. df.pivot(index='id', columns='var', values='val') goes from long to wide but requires unique index/column pairs; when duplicates exist, df.pivot_table(index='id', columns='var', values='val', aggfunc='sum') aggregates them (defaulting to mean). The reverse, df.melt(id_vars=['id'], var_name='var', value_name='val'), turns wide columns into long rows. stack() compresses a level of the column index into the row index, yielding a Series with a MultiIndex, and unstack() does the opposite. Cross-tabulations are produced with pd.crosstab(df['a'], df['b']), with optional values= and aggfunc= to summarize another column. Finally, continuous values can be binned into discrete buckets with pd.cut(df['v'], bins=[0, 10, 20, 100]), which returns a categorical Series, or with pd.qcut(df['v'], q=4) for equal-frequency quartiles.

Date handling is a first-class concern in Pandas. Strings become datetimes with pd.to_datetime(df['date_str']), optionally sped up by passing an explicit format='%Y-%m-%d'. Once a column holds datetimes, the .dt accessor exposes components: df['date'].dt.year, .dt.month, .dt.day, .dt.dayofweek, and .dt.hour. Promoting a datetime column to the index with df.set_index('date', inplace=True) turns the frame into a time series and unlocks partial-string slicing such as df.loc['2024-01']. Resampling at a different frequency is then a single call: df.resample('M').mean() aggregates monthly, with 'D', 'W', 'Q', 'Y', and 'H' as other common aliases. Time-based rolling windows like df.rolling('7D').sum() compute over a calendar-aware window, distinct from the row-count window of df.rolling(window=3).mean(). df['v'].expanding().mean() computes the running mean from the start up to each row.

Time-series analysis often needs lagged and differenced values. df['v'].shift(1) shifts the series down so each row sees the previous value; shift(-1) shifts up. df['v'].diff() subtracts the prior row from the current one (the first row is NaN, and diff(periods=k) uses a larger lag), while df['v'].pct_change() computes \( (v_t - v_{t-1}) / v_{t-1} \). For locating extrema, df.idxmax() returns the index label of the maximum per column, and df['col'].argmax() the integer position (deprecated in Pandas 2.0; the recommended alternative is df['col'].to_numpy().argmax()). For detecting outliers with the IQR rule, the bounds are \( q_1 - 1.5 \cdot \text{IQR} \) and \( q_3 + 1.5 \cdot \text{IQR} \), where \( \text{IQR} = q_3 - q_1 \), and the boolean mask is \( (v < q_1 - 1.5 \cdot \text{IQR}) \,|\, (v > q_3 + 1.5 \cdot \text{IQR}) \).

The .str accessor turns string columns into vectorized string operations. Splitting on a delimiter produces multiple columns with df['col'].str.split(',', expand=True); regex capture groups are pulled out by df['col'].str.extract(r'(\d+)'); whitespace is removed with df['col'].str.strip() (with lstrip and rstrip variants); substrings are replaced by df['col'].str.replace('old', 'new', regex=False); length is queried with df['col'].str.len(); and prefix or suffix checks are written as df['col'].str.startswith('pre') (also endswith and contains). Two columns can be concatenated into one with df['full'] = df['first'] + ' ' + df['last'] or with df['full'] = df[['first', 'last']].agg(' '.join, axis=1). For low-cardinality string columns, the categorical dtype is worth using: df['c'] = df['c'].astype('category') saves memory and speeds up groupby, and ordering can be enforced with pd.Categorical(df['c'], categories=['low', 'med', 'high'], ordered=True).