datafusion-python

DataFusion Python DataFrame API Guide

What Is DataFusion?

DataFusion is an in-process query engine built on Apache Arrow. It is not a database -- there is no server, no connection string, and no external dependencies. You create a SessionContext, point it at data (Parquet, CSV, JSON, Arrow IPC, Pandas, Polars, or raw Python dicts/lists), and run queries using either SQL or the DataFrame API described below.

All data flows through Apache Arrow. The canonical Python implementation is PyArrow (pyarrow.RecordBatch / pyarrow.Table), but any library that conforms to the Arrow C Data Interface can interoperate with DataFusion.

Core Abstractions

Abstraction	Role	Key import
SessionContext	Entry point. Loads data, runs SQL, produces DataFrames.	from datafusion import SessionContext
DataFrame	Lazy query builder. Each method returns a new DataFrame.	Returned by context methods
Expr	Expression tree node (column ref, literal, function call, ...).	from datafusion import col, lit
functions	290+ built-in scalar, aggregate, and window functions.	from datafusion import functions as F
functions.spark	PySpark-compatible function surface (parameter names match pyspark.sql.functions).	from datafusion.functions import spark

Import Conventions

from datafusion import SessionContext, col, lit
from datafusion import functions as F
from datafusion.functions import spark   # only when porting pyspark code

Data Loading

ctx = SessionContext()

# From files
df = ctx.read_parquet("path/to/data.parquet")
df = ctx.read_csv("path/to/data.csv")
df = ctx.read_json("path/to/data.json")

# From Python objects
df = ctx.from_pydict({"a": [1, 2, 3], "b": ["x", "y", "z"]})
df = ctx.from_pylist([{"a": 1, "b": "x"}, {"a": 2, "b": "y"}])
df = ctx.from_pandas(pandas_df)
df = ctx.from_polars(polars_df)
df = ctx.from_arrow(arrow_table)
df = ctx.read_batch(record_batch)               # one pa.RecordBatch, no named table
df = ctx.read_batches([batch1, batch2])         # several pa.RecordBatch

# From SQL
df = ctx.sql("SELECT a, b FROM my_table WHERE a > 1")