accelerating-python

Accelerating Python

Rule: profile first; query before loading; then pick the smallest acceleration that matches the bottleneck. Most "slow Python" and "data too big" problems are solved before you reach for multiprocessing or a GPU.

Order of operations

1. Measure the slow part — don't guess.
2. Reduce data with filters/projections before loading (push them into the file reader).
3. Use better engines: DuckDB, Polars, Arrow, NumPy — they stream and push down.
4. Vectorize simple array/dataframe operations.
5. Numba for tight numeric loops that don't vectorize cleanly.
6. Multiprocessing for CPU-bound Python — see parallel python.
7. GPU only for GPU-shaped work — see using GPUs.

Do not start at step 6 if the real bottleneck is CSV parsing, GPFS metadata, a database query, or network latency.

Quick profiler

import cProfile
import pstats

with cProfile.Profile() as profiler:
    main()

stats = pstats.Stats(profiler).sort_stats("cumtime")
stats.print_stats(25)

For a running job, py-spy dump --pid PID is often more useful (if installed).

Store data in a columnar format

• Parquet for tabular data, Arrow datasets for partitioned data, HDF5 for array-like data.
• Avoid thousands of CSVs, repeated CSV parsing, and Excel as an intermediate format.

One reused Parquet beats 10k CSVs for both speed and GPFS metadata health.

Query before loading

The single biggest win for both speed and memory: filter and project in the reader so the full table never enters Python. DuckDB (SQL) and Polars (lazy) both push predicates/projections into the Parquet scan and stream the result — pick whichever idiom matches your code.

DuckDB — SQL over Parquet with pushdown:

import duckdb

duckdb.sql("""
COPY (
  SELECT gvkey, fyear, sales
  FROM read_parquet('/gpfs/project/myproject/data/raw/panel/*.parquet')
  WHERE fyear >= 2010
) TO '/gpfs/project/myproject/data/derived/sales.parquet'
(FORMAT PARQUET)
""")