make-pythonic

Make Python API Functions More Pythonic

You are improving the datafusion-python API to feel more natural to Python users. The goal is to allow functions to accept native Python types (int, float, str, bool, etc.) for arguments that are contextually always or typically literal values, instead of requiring users to manually wrap them in lit().

Core principle: A Python user should be able to write split_part(col("a"), ",", 2) instead of split_part(col("a"), lit(","), lit(2)) when the arguments are contextually obvious literals.

Scope: functions vs functions.spark

Both python/datafusion/functions/__init__.py and python/datafusion/functions/spark.py are in scope. We want both to feel pythonic — accept native Python types where the argument is contextually a literal — but functions.spark carries an additional constraint: every signature must remain compatible with pyspark.sql.functions.

Compatibility rules for the spark namespace:

• Parameter names must match pyspark exactly. Pyspark callers pass by keyword (spark.shiftleft(col=..., numBits=...)), so renames break them. Do NOT rename a parameter just because it would be more pythonic in the main namespace.
• Positional order must match pyspark exactly. Reordering breaks positional pyspark calls.
• Type unions may widen the input set, never narrow it. Pyspark accepts Column or str (column name) for most args; we accept Expr already, and widening to Expr | int / Expr | str for literal-friendly arguments is on-brand because the int/str case is exactly what a pyspark caller would also try. Just verify the widened set is a superset of what pyspark accepts for that arg.
• Extra keyword arguments are allowed as long as they default to None and pyspark's positional/keyword form still works (e.g. the spark avg/try_sum/collect_list/collect_set retain DataFusion's distinct/filter/order_by/null_treatment kwargs).