dataframe-performance

DataFrame Performance

Overview

Most slow data code is Python looping over rows, reading columns it never uses, or holding strings as generic objects. Fix those in the library the repository already uses before reaching for a new one; when the data truly outgrows memory, DuckDB and Polars process it lazily and out of core.

Core principle: measure first, pin behaviour with a test, optimise, measure again — and report both numbers.

1. Measure, then pin

start = time.perf_counter()
result = build_features(sample)
print(f"{time.perf_counter() - start:.2f}s, {result.memory_usage(deep=True).sum() / 1e6:.0f} MB")

Measure on a realistic sample (not 20 rows, not the full warehouse). Before rewriting a slow function, make sure a fixture test pins its output; the faster version must pass the same test. Put "before → after" timings and memory in your closing message.

2. pandas 3 semantics you must write for

pandas 3 makes Copy-on-Write the only mode and a dedicated string dtype the default. Code written for pandas 1.x/2.x can silently stop working:

# ❌ chained assignment — never updates df under Copy-on-Write
df["discount"][df["plan"] == "pro"] = 0.1
# ❌ inplace on a selected column — never updates df
df["age"].fillna(0, inplace=True)

# ✅ one indexing operation on the object you mean to change
df.loc[df["plan"] == "pro", "discount"] = 0.1
df["age"] = df["age"].fillna(0)

• A filtered frame behaves as a copy: sub = df[mask]; sub["x"] = 1 changes sub only, with no SettingWithCopyWarning and no .copy() needed.
• Strings default to the str dtype (Arrow-backed when PyArrow is installed; missing values are NaN). df["s"].dtype == object is now False — use pd.api.types.is_string_dtype.
• The repository's pinned pandas wins; if it is still on 2.x, the patterns above are also the correct 2.x style.

3. Vectorise instead of looping

# ❌ Python function per row
df["tier"] = df.apply(lambda r: "high" if r.spend > 1000 else ("mid" if r.spend > 100 else "low"), axis=1)

# ✅ vectorised
df["tier"] = np.select([df["spend"] > 1000, df["spend"] > 100], ["high", "mid"], default="low")
# ✅ or as an ordered categorical
df["tier"] = pd.cut(df["spend"], bins=[-np.inf, 100, 1000, np.inf], labels=["low", "mid", "high"])