Programming for Data Processing

Lectures: Nguyễn Tuấn Phong, Đào Việt Anh

Labs: Trịnh Ngọc Huỳnh, Phạm Tiến Du

Semester: 2025-2026-1

Institute for AI, VNU-UET

Lecture 10
Part 2:
Pandas Aggregation & Groupby

Split–Apply–Combine

  • Split by keys → Apply functions → Combine
  • Aggregation vs Transformation
  • Manage index with as_index

Groupby basics

  • Use groupby() + reducers
  • Prefer as_index=False for tidy outputs
  • Set categorical dtypes for speed/size

Named aggregations

  • One pass, many stats
  • Clear, labeled metrics
  • Compute first, format later

Custom functions (caution)

  • Prefer vectorized reductions
  • Functions must be pure (no side effects)
  • Be mindful of performance with apply

Groupwise features w/ transform

  • Same-length output aligned to rows
  • Examples: share-of-total, z-scores
  • Great for features in modeling

When (not) to use apply

  • Use for per-group DataFrame logic
  • Disable keys: group_keys=False
  • Beware memory/parallel limits

Pivoting & MultiIndex

  • Use pivot_table (handles duplicates)
  • margins=True for totals
  • Clean axes before export

Time grouping & resampling

  • pd.Grouper(key="ts", freq="W")
  • resample("D") on DatetimeIndex
  • Careful with timezones

Rolling windows

  • rolling(window=7, min_periods=1)
  • Use for smoothing, thresholds, alerts
  • Validate with known totals