emmy AGENTS.md

AGENTS.md

This file provides guidance to coding agents when working with code in this repository.

Project Overview

Emmy is a Python tool for deploying and benchmarking LLM inference on GPU servers. It supports vLLM and SGLang engines, provides a CLI for local and remote (SSH) deployment of models via Docker Compose, plus automated benchmarking across multiple servers.

README.md is the canonical project overview and architecture index. Read it first, then use its links to locate the relevant subsystem documentation. Do not duplicate the architecture index in this file.

When the user asks about a CLI flag, recipe field, or matrix combinator, use the README index to find and read the relevant ARCHITECTURE.md before answering.

Prerequisites

• Python 3.12+ with venv
• make setup to create the virtual environment and install dependencies
• Docker and Docker Compose for local deployments
• HF_TOKEN environment variable for HuggingFace model downloads
• EMMY_DUMP_DIR environment variable (optional) — when set, compiler stages dump intermediate debug artifacts (graphs, CUDA kernels, execution plans) to this directory. Frontend provenance slices used by tune --bench stay in memory; stable Torch IR is persisted only inside golden YAML. Kernels are named after the operations they realize (k_rms_norm, k_sdpa_reduce).
• EMMY_FREEZE_DIR environment variable (optional) — overrides the measurement freeze the prior is evaluated against (emmy eval prior --dataset nodes, and emmy fit's measured cells). Defaults to the repo-checked emmy/compiler/pipeline/search/freezes/ — a digest-pinned, version-stamped snapshot that is identical on every machine, which is what makes a reported prior number reproducible. The tune DB and the online reservoir are machine-local and mutable; reach them with --db when you want one machine's data, not as the default. The freeze's payload YAML is tracked in git LFS; its manifest is plain git so provenance stays diffable. Re-freeze with scripts/freeze_node_store.py.
• EMMY_TUNE_DB environment variable (optional) — overrides the default tuning SQLite cache path (~/.cache/emmy/autotune.db). emmy tune reads from / writes to this path. NOTE: greedy compile / run resolve forks through the deploy evidence hierarchy — the live card's recorded goldens first (the repo-shipped verified tier; consulted, never trained on), then measured reservoir/DB evidence, then the global Prior (the online prior with its offline cold-start fallback; the old _best_fork DB→fork replay was removed). The online prior is a separate JSON checkpoint (EMMY_ONLINE_FILE → ~/.cache/emmy/online.json; legacy EMMY_PRIOR_FILE still accepted) that tune writes and compile / run read. Use the README architecture index for the prior and two-level autotune design.