powerbi-spark-livy

Executing Spark on Fabric without a notebook

The Livy endpoint on a Fabric lakehouse runs arbitrary PySpark or Python on Fabric Spark compute as an ephemeral session — no notebook artifact is created, nothing is persisted in the workspace, and the session is yours to delete. It is the right tool for profiling a Delta table, checking a silver-layer row count, validating a transform before it becomes a pipeline, or answering "is the data even there" without leaving a notebook behind in a governed workspace.

It is the wrong tool for anything scheduled or reviewed. If the code should run again tomorrow, it belongs in a notebook or a Spark job definition under source control — powerbi-fabric-pipelines.

Prerequisites

Requirement	Check
Azure CLI signed in	az account show
A lakehouse in the target workspace	the session runs against it; fab ls "<ws>.Workspace"
Fabric capacity (F SKU or trial)	Spark draws CU — see below
Workspace Contributor or above	session creation is a write operation

Authentication — the one thing people get wrong

The Livy API needs a token scoped to the Fabric service, acquired through Azure CLI:

import json
import subprocess

result = subprocess.run(
    ["az", "account", "get-access-token", "--resource", "https://api.fabric.microsoft.com"],
    capture_output=True,
    text=True,
    check=True,
)
token = json.loads(result.stdout)["accessToken"]

Two rules:

• A fab auth token is not interchangeable here. It will authenticate the call and then fail on OneLake storage access from inside the Spark session, which surfaces as a confusing permission error on the first spark.read, not on session creation.
• Never print, log or echo the token, and never pass it on a command line — process arguments are readable by other users on the host. Pass it in the Authorization header from in-process variables only.

Session lifecycle

1. POST   .../livyapi/versions/<ver>/sessions                 {"kind": "pyspark"}
2. GET    .../sessions/{id}                 poll until state == "idle"   (~30–90 s cold)
3. POST   .../sessions/{id}/statements      {"code": "...", "kind": "pyspark"}
4. GET    .../sessions/{id}/statements/{n}  poll until state == "available"
5. DELETE .../sessions/{id}                 ALWAYS — in a finally block