emmy: Instructions file for Codex

AGENTS.md

This file provides guidance to coding agents when working with code in this repository.

Communication

Write to the user the way you would speak to them.

• Short sentences. One idea per sentence.
• Say the few things that matter most. Leave out the rest unless the user asks.
• No file paths, symbol names, or code unless the user asks for them.
• Simple technical English. Common words over rare ones.
• Only established vocabulary: GLOSSARY.md terms, other standard repo or field terms, or plain language. Never invent a label.
• Answer first. Add background only when it changes what the user does next.
• Disagree when you have reason to. Say what is wrong and what you would do instead. The goal is the right answer, not agreement.
• Raise a concern or change the direction of the conversation only when the stake is real: a wrong approach, a risk, a much better option. Small things are not worth the interruption. Judge it the way one developer would judge it when talking to another.

Project Overview

Emmy is a Python tool for deploying and benchmarking LLM inference on GPU servers. It supports vLLM and SGLang engines, provides a CLI for local and remote (SSH) deployment of models via Docker Compose, plus automated benchmarking across multiple servers.

README.md is the canonical project overview and architecture index. Read it first, then use its links to locate the relevant subsystem documentation. Do not duplicate the architecture index in this file.

When the user asks about a CLI flag, recipe field, or matrix combinator, use the README index to find and read the relevant ARCHITECTURE.md before answering.

Prerequisites

• Python 3.12+ with venv
• A Rust toolchain (cargo): compiled programs execute through the Rust runtime, which make setup builds into the package as the emmy.emmy_runtime extension; use the README architecture index for the runtime's design.
• make setup to create the virtual environment and install dependencies
• Docker and Docker Compose for local deployments
• HF_TOKEN environment variable for HuggingFace model downloads
• EMMY_DUMP_DIR environment variable (optional) — when set, compiler stages dump intermediate debug artifacts (graphs, CUDA kernels, execution plans) to this directory. Frontend provenance slices used by run --bench stay in memory; stable Torch IR is persisted only inside golden files. Kernels are named after the operations they realize (k_rms_norm, k_sdpa_reduce).
• EMMY_FREEZE_DIR environment variable (optional) — the directory a run --bench --tune N freezes its rows into as a file beside the card's: the repository freeze of an editable checkout when unset, none when set empty (the test suite), else that directory. README "Keep what a tune measured".
• EMMY_TUNE_DB environment variable (optional) — overrides the default tuning SQLite cache path (~/.cache/emmy/autotune.db), the store of measured kernel rows. emmy run --bench writes to it, and a greedy compile / run / serve creates it on first use: the golden rows in scope (the live card's repository goldens, or the file --golden PATH names) are imported into it before the compile picks, once per golden digest. NOTE: those commands resolve forks through ONE measured-evidence pick — this DB's perf rows and the golden rows among them, ranked fastest-first; a kernel-set decision (a routing row, a golden's cut or split) is priced as the sum of its pieces' rows; and the priors (fit by emmy fit: the schedule prior weights/schedule.json for a kernel's schedule rows, the placement prior weights/placement.json for a kernel-set fork's arms) decide only where nothing was measured; --strict-evidence turns that fall-through into an error. run --golden PATH --bench writes what it measures back into this DB (--record / --record-greedy write the golden file too), which is how a golden row becomes what the next compile picks. Use the README architecture index for the prior and the evidence design.