agentops-open-course: Instructions file for Codex

AGENTS.md

Guidance for coding agents working in the AgentOps Open Course. Humans should start with README.md. This repository dogfoods the AGENTS.md convention taught in Chapter 1.

Repository purpose

The course teaches the complete lifecycle of one AgentOps Agent with Google ADK, agentgateway, kagent, MLflow, and OpenTelemetry. The completed executable reference accompanies cumulative learner-owned Python exercises; it must not drift into a collection of illustrative snippets. Chapter 8.7 turns that reference into a capstone contract for a learner-owned domain.

• docs/ contains FAQ-based course pages published by Zensical.
• agents/python/ is the locked Python reference agent, offline tests, and model-backed evaluations.
• agents/data/ is immutable seed input: SQLite, logs, runbooks, and the agent's runtime Agent Skills.
• skills/ holds portable Agent Skills (npx skills add …), validated by mise run check:skills; runtime skills live separately under agents/data/skills.
• clients/web/ is a minimal, offline, dependency-free A2A web client for the AgentOps Agent.
• load/ holds k6 load tests and the documented latency budgets for the platform.
• infra/agentgateway/{host,k3d,gke}/ contains the three data-plane profiles.
• infra/k8s/base plus infra/k8s/overlays/{local,local-gemini,gke} contains the shared Kubernetes deployment.
• infra/kagent/ declares the BYO Agent, gateway ModelConfig, and governed RemoteMCPServer.
• infra/mlflow/ builds the locked non-root MLflow server.
• infra/observability/ contains host Compose and in-cluster OTel/Prometheus/Grafana resources.
• infra/gcp/ is a plan-first OpenTofu module for the optional GKE lab.

Course invariants

• Docs mirror source. Critical Python excerpts use checked pymdownx.snippets regions from agents/python; commands/manifests match infra. Prefer a short exact excerpt plus a source link over a second pseudo-implementation.
• Every course page is an FAQ. It starts with YAML description front matter, contains at least one H2, and every H2 ends in ?. scripts/check_conventions.py enforces this and the page frame below.
• Seed and state stay separate. agents/data/incidents.db is never mutated. Host writes go to agents/python/.state; Kubernetes agent/MCP processes share agentops-agent-state so reads remain coherent with approved writes. Only the A2A startup and direct write boundary may prepare or migrate runtime state; probes and read tools stay read-only.
• Restore is crash-recoverable, not an instantaneous multi-file rename. Stop every writer first. agent.state serializes backup/restore with a process lock and fsyncs a three-phase journal; A2A startup recovers an interrupted transaction before schema preflight or publication. Never bypass that boundary with direct file copies or delete unexplained .restore-* evidence.
• Reads and writes have different authority. The conversational entrypoint alone switches its six read/runbook tools from direct local calls to MCP through AGENT_MCP_URL; workflow and coordinator specialists always bind local tools. The MCP toolset passes tool_filter=MCP_READ_TOOL_NAMES (mcp_client.py), so a server cannot widen the surface by advertising more tools. restart_service and resolve_incident remain in-process, require ADK confirmation, validate targets, and append audit evidence in the same transaction. Replays with the same invocation, action, and target return the original audit row without mutating state again.
• Policy is attached once, at the app boundary. AgentOpsPolicyPlugin in governance.py is registered on the App exported by composition.py and src/agent/__init__.py. ADK discovery prefers app over root_agent. Hooks cover every agent and workflow node; never add per-agent callback lists. Before-model order is budget → compaction → redaction; the first non-None return short-circuits. ADK's stock evaluator appends its plugins after the App policy (a bare agent before ADK 2.10): use evals/governed_adk_eval.py for ADK evals, which keeps evaluator evidence first and the policy exactly once, and InMemoryRunner(app=build_app(...)) for MLflow.
• Skills and retrieved data have different trust. The carve-out is keyed on the ADK LoadSkillTool type, which only the locally built skill_toolset() constructs — not on the tool's name, which any MCP server could claim. That result is reviewed repository instruction, so it bypasses injection neutralization and spotlighting while retaining recursive PII/credential redaction. Every other tool result stays data-hardened by default.
• Audit is append-only, not immutable. Every row carries its audit schema version. SQLite triggers block row update/delete through the schema; administrators can still alter the file/schema. Do not overclaim.
• Telemetry content stays private by default. Both ADK/GenAI content-capture variables default to literal false. PII callbacks cover outbound model requests, inbound model responses, and tool output, but raw session ingestion occurs earlier.
• No LiteLLM or garak contract. Runtime/evaluation uses ADK's OpenAI-compatible client for Ollama/agentgateway or native Gemini by default. mise run redteam is deterministic offline adversarial regression, not live-model penetration testing.
• Planning is bounded. root_agent plans only multi-step investigations and verifies approved actions afterward. triage_workflow is the runnable, read-only plan → investigate → evidence review → recommend path; do not replace it with an unbounded reflection loop.
• Cost-efficient by default. Prefer deterministic offline tests and fakes, the smallest model that can validate the behavior, and single-replica resource-bounded local services. Measure before increasing model size, context, RAM, CPU, storage, replicas, or load-test concurrency. Do not start a cluster, observability stack, model server, paid API, or cloud resource unless it materially validates the current boundary; stop temporary processes and tear down disposable resources when the check is complete.