Skip to content

Chapter 7 of 7

Time Series, Strings, and Categorical Types

Date handling is a first-class concern in Pandas. Strings become datetimes with pd.to_datetime(df['date_str']), optionally sped up by passing an explicit format='%Y-%m-%d'. Once a column holds datetimes, the .dt accessor exposes components: df['date'].dt.year, .dt.month, .dt.day, .dt.dayofweek, and .dt.hour. Promoting a datetime column to the index with df.set_index('date', inplace=True) turns the frame into a time series and unlocks partial-string slicing such as df.loc['2024-01']. Resampling at a different frequency is then a single call: df.resample('M').mean() aggregates monthly, with 'D', 'W', 'Q', 'Y', and 'H' as other common aliases. Time-based rolling windows like df.rolling('7D').sum() compute over a calendar-aware window, distinct from the row-count window of df.rolling(window=3).mean(). df['v'].expanding().mean() computes the running mean from the start up to each row.

Time-series analysis often needs lagged and differenced values. df['v'].shift(1) shifts the series down so each row sees the previous value; shift(-1) shifts up. df['v'].diff() subtracts the prior row from the current one (the first row is NaN, and diff(periods=k) uses a larger lag), while df['v'].pct_change() computes \( (v_t - v_{t-1}) / v_{t-1} \). For locating extrema, df.idxmax() returns the index label of the maximum per column, and df['col'].argmax() the integer position (deprecated in Pandas 2.0; the recommended alternative is df['col'].to_numpy().argmax()). For detecting outliers with the IQR rule, the bounds are \( q_1 - 1.5 \cdot \text{IQR} \) and \( q_3 + 1.5 \cdot \text{IQR} \), where \( \text{IQR} = q_3 - q_1 \), and the boolean mask is \( (v < q_1 - 1.5 \cdot \text{IQR}) \,|\, (v > q_3 + 1.5 \cdot \text{IQR}) \).

The .str accessor turns string columns into vectorized string operations. Splitting on a delimiter produces multiple columns with df['col'].str.split(',', expand=True); regex capture groups are pulled out by df['col'].str.extract(r'(\d+)'); whitespace is removed with df['col'].str.strip() (with lstrip and rstrip variants); substrings are replaced by df['col'].str.replace('old', 'new', regex=False); length is queried with df['col'].str.len(); and prefix or suffix checks are written as df['col'].str.startswith('pre') (also endswith and contains). Two columns can be concatenated into one with df['full'] = df['first'] + ' ' + df['last'] or with df['full'] = df[['first', 'last']].agg(' '.join, axis=1). For low-cardinality string columns, the categorical dtype is worth using: df['c'] = df['c'].astype('category') saves memory and speeds up groupby, and ordering can be enforced with pd.Categorical(df['c'], categories=['low', 'med', 'high'], ordered=True).

All chapters
  1. 1Foundations and I/O
  2. 2Inspecting, Selecting, and Filtering
  3. 3Modifying and Sorting
  4. 4Aggregation and Apply
  5. 5Missing Data and Duplicates
  6. 6GroupBy, Combining, and Reshaping
  7. 7Time Series, Strings, and Categorical Types

Drill it

Reading is not remembering. These come from the Pandas Dataframe Operations deck:

Q

What is a Pandas DataFrame?

A two-dimensional, size-mutable, tabular data structure with labeled axes (rows and columns), similar to a spreadsheet or SQL table.

Q

What library provides DataFrame in Python?

Pandas (import pandas as pd).

Q

How do you create a DataFrame from a dictionary of lists?

pd.DataFrame({'col1': [1, 2], 'col2': [3, 4]})

Q

How do you create a DataFrame from a list of dictionaries?

pd.DataFrame([{'a': 1}, {'a': 2, 'b': 3}]); missing keys become NaN.