pyspark-datagen AGENTS.md

Agent guide: learning-spark-datagen

Quick reference for AI agents (and humans) so you don’t have to scan the README or multiple files to run data generation and Delta workflows.

Agent rules and skills (load these automatically)

Project-specific rules and skills live under .agents/ (not .cursor/). This AGENTS.md is loaded automatically, so at the start of any task in this repo, also load and follow the following:

• Rules — read every file in .agents/rules/ and apply them as always-on guidance:
  • .agents/rules/agents-and-tests.mdc — when you find a better approach, update AGENTS.md, add/extend unit tests, and fix one problem at a time.
  • .agents/rules/git-branching.mdc — never commit directly to main; before starting work, check the current branch and (if on main) create a feature branch first. Use AskQuestion to offer a suggested feat/<short-description> name or let the user provide their own. Prefixes: feat/, fix/, chore/, refactor/, test/. One logical unit of work per branch.
• Skills — when a task matches a skill's description, read its SKILL.md first and follow it:
  • .agents/skills/add-datagen-generator/SKILL.md — add a new Protobuf-backed data generator (new proto, GenX class, CLI, tests) following the User/Order pattern.

If you add new rules or skills, drop them under .agents/rules/ or .agents/skills/<name>/SKILL.md and add a bullet here so they continue to load automatically.

Where to run

All commands below assume you’re in learning-spark-datagen/ (the package with main.py and pyproject.toml). Use uv run to execute the CLI.

cd learning-spark-datagen

CLI: main.py

Entry point: uv run main.py (or uv run main.py --generate ...).

Flag	Purpose
--generate	Enable generate mode (required for data gen).
--type users | orders | products | sessions | rain_sensors	Entity to generate (default: users).
--count N	Number of records — or sessions for --type sessions — (default: 100).
--output PATH	File path for JSON/NDJSON; required for --format delta (Delta table directory).
--format json | delta	json = NDJSON to file (or stdout if --output omitted); delta = Delta table. Auto-defaults to delta when --users-table or --products-table is set; otherwise json.
--users-file PATH	For --type orders and --type sessions. NDJSON file of users; user_id will be one of these UUIDs.
--users-table DELTA_DIR	Alternative to --users-file; reads users from a Delta table via Spark. Mutually exclusive with --users-file.
--products-file PATH	For --type sessions only. NDJSON file of products; every event's items[].item_id will reference these products.
--products-table DELTA_DIR	Alternative to --products-file; reads the product catalog from a Delta table via Spark. Mutually exclusive with --products-file.
--start-date YYYY-MM-DD	For --type sessions only. Earliest session date (default 2026-01-01).
--end-date YYYY-MM-DD	For --type sessions only. Latest session date (default 2026-05-28).
--sessions-per-user-alpha FLOAT	For --type sessions only. Zipf exponent for user picking (default 1.5; higher = more skew toward loyal customers).
--funnel-profile realistic | aggressive | conservative	For --type sessions only. Built-in funnel drop-off profile (default realistic).
--seed N	Random seed (default: 42).