Skip to content

Chapter 4 of 7

Aggregation and Apply

The quickest overview of a numeric frame is df.describe(), which returns count, mean, standard deviation, min, quartiles, and max for every numeric column. For individual reductions, df.mean() computes the mean of each column, df.count() returns non-null counts per column, and df['col'].nunique() counts distinct values in a column. The distinct values themselves can also be inspected with df['col'].unique(), which returns a NumPy array, or with df['col'].value_counts(), which produces a frequency Series sorted in descending order by default.

Aggregation is governed by the axis argument. With the default axis=0, a function operates down the rows and produces one value per column; with axis=1, it operates across the columns and produces one value per row. df.apply(np.sum) therefore sums each column, while df.apply(np.sum, axis=1) sums each row. For element-wise work, df.applymap(func) (renamed df.map() in Pandas 2.1 and later) hits every cell, and on a Series both df['col'].map(func) and df['col'].apply(func) do the same per element, with the extra ability for map to accept a dict or Series for value substitution. To rewrite values globally, df.replace({'old1': 'new1', 'old2': 'new2'}) maps old values to new ones.

Several helpers complete the analysis toolbox. df['col'].rank(method='average') assigns ranks, with average, min, max, first, and dense as the available tie-breaking methods. df.corr(numeric_only=True) and df.cov(numeric_only=True) return the pairwise correlation and covariance matrices of the numeric columns. Standardizing a column to a z-score is a one-liner: (df['v'] - df['v'].mean()) / df['v'].std(). For sampling, df.sample(n=5) draws five random rows and df.sample(frac=0.1, random_state=42) takes 10%; passing random_state makes the draw reproducible. To clip values to a sensible range, df['v'].clip(lower=0, upper=100) caps them without dropping them. Iterating over rows is possible with for index, row in df.iterrows(): ..., which is convenient but slow; for row in df.itertuples(): ... is the faster namedtuple alternative, and vectorized operations should be preferred whenever feasible.

All chapters
  1. 1Foundations and I/O
  2. 2Inspecting, Selecting, and Filtering
  3. 3Modifying and Sorting
  4. 4Aggregation and Apply
  5. 5Missing Data and Duplicates
  6. 6GroupBy, Combining, and Reshaping
  7. 7Time Series, Strings, and Categorical Types

Drill it

Reading is not remembering. These come from the Pandas Dataframe Operations deck:

Q

What is a Pandas DataFrame?

A two-dimensional, size-mutable, tabular data structure with labeled axes (rows and columns), similar to a spreadsheet or SQL table.

Q

What library provides DataFrame in Python?

Pandas (import pandas as pd).

Q

How do you create a DataFrame from a dictionary of lists?

pd.DataFrame({'col1': [1, 2], 'col2': [3, 4]})

Q

How do you create a DataFrame from a list of dictionaries?

pd.DataFrame([{'a': 1}, {'a': 2, 'b': 3}]); missing keys become NaN.