in-cluster-observability: Instructions file for Codex

AGENTS.md

This file documents conventions and operational context for working in this repository. Both humans and agentic coding tools should read it before any non-trivial change. Keep it current as conventions evolve.

GEMINI.md is a stub that points back here so the two tools see the same content.

Current state

This repo is in the middle of a planned rewrite. The legacy POC code (Prometheus + eBPF agent at the root, OpenTelemetry sink/query pipeline under opentelemetry/, the obs/ logging library) was removed early in the rewrite. POC code is preserved on main and reachable via git log main -- <path>.

All code lives at the repo root as a single AP root and single Go module github.com/gke-labs/in-cluster-observability (per ADR-0015). Per ADR-0018, OBI runs as a sibling container in the agent DaemonSet pod, not as an embedded Go library — our agent is an OTLP receiver + OBI config writer.

Milestone status:

• v0.1 Foundation (#64–#69) landed: AP root, public package skeletons, OBI adapter shell, container image, minimal DaemonSet.
• v0.2 Capture MVP (#70–#77) landed: OTLP receivers (gRPC + HTTP loopback), OBI config writer, AllowPID/BlockPID with reload coalescer, L4 TCP + HTTP/1.1 translation, OTel self-obs metrics, panic recovery + ModuleDegraded events, debug HTTP endpoint, contract-test harness. DaemonSet now has two containers (obi sibling + agent).
• v0.3 (lean) — agent + OBI native enrichment (per ADR-0021, supersedes the original v0.3 "Storage MVP" plan). Adds: OBI v0.9 schema fixes in internal/obiconfig (discovery.instrument + open_ports string + target_pids), --obi-instrument-ports smoke-test seed, OTel SDK Prometheus exporter always-on with a forwarder that re-records OBI's translated metrics, --scrape-addr agent listener at :9090, RBAC for OBI's K8s metadata informer, DaemonSet refactor (right caps + OTEL_EBPF_CONFIG_PATH env var + /var/run/obi + /sys/fs/cgroup mounts). What we explicitly did not build: a separate internal/pidcache, internal/enricher, pkg/sink/promscrape, or pkg/store.MetricStore — OBI's K8s informer + the OTel SDK + this single forwarder replace all of them.
• v0.4 Control Plane MVP in progress (per ADR-0022). Ships across 4 stacked phase PRs: design refresh + ADR (Phase 0), CRD types + gRPC stubs (Phase 1), reconciler + stream + leader election (Phase 2), RBAC + CR status + AgentStatus (Phase 3). Decisions pinned: API group ollie.gke-labs.dev, controller framework sigs.k8s.io/controller-runtime, codegen controller-gen only, validating webhook deferred to v0.5, identity broadcasting cut (OBI's informer covers source-side per ADR-0021). New binary cmd/ollie-controller/; new packages pkg/controller/{api/v1alpha1,pb,reconciler,stream,agentclient}/; new manifests k8s/{controller.yaml,controller-rbac.yaml,crds/}.
• v0.4.5 Verification & Soundness landed (per ADR-0023, issues #150–#158, PRs #160–#167). The OBI boundary is now CI-checkable and the metrics path is sound: ap-e2e Kind presubmit exercising the real DaemonSet on every PR; contract fixtures recorded from the pinned OBI image (recorder in tests/contract/obi/record_test.go, provenance in testdata/translation/RECORDED.md), goldens freeze metric type/temporality/bucket layout; the agent re-emits OBI metrics via a const-metric Prometheus collector (cmd/ollie/forward.go) — cumulative totals pass through (OBI exports cumulative; the old forwarder inflated counters), histograms keep buckets, stale series evicted; pkg/capture lifecycle bugs fixed; DaemonSet production trim (PSA labels, tolerations, system-node-critical, tcpSocket probes, seccomp; CAP_SYS_ADMIN is empirically required by OBI's L7 path on v0.9.0 and v0.10.0 — re-attempt each bump); OBI image now otel/ebpf-instrument:v0.10.0 (no breaking wire changes; additive queue/processing sub-spans).
• v0.5 Sinks + Query + HPA landed (milestone closed 2026-07-29; vertical slice PRs #170–#173, breadth PRs #174–#178). An HPA scales on OBI-captured metrics end to end, and every ADR-0024 egress surface ships: scrape, PromQL API + custom-metrics + remote read, OTLP push relays + remote-write, CEL streaming subscribe, iobsctl. The #157 gate had resolved as ADR-0024: extensibility via wire protocols, not a Go library — pkg/sink, pkg/obsapi, pkg/store, pkg/query, and pkg/topology are deleted; store/query land under internal/ shaped by the HPA slice. Implementation decisions pinned in ADR-0025 (vertical slice: tsdb.Open, registry self-scrape ingest, storage-layer fan-out via remote read with central PromQL, cmd/ollie-query, headless-Service discovery, hand-rolled custom-metrics API, self-signed TLS until v0.6 — landed as PRs #170–#173) and ADR-0026 (breadth: raw-OTLP tee on the bridge feeding a spans-only ring + OTLP relay egress + remote-write; agent-side CEL for streaming subscribe; iobsctl over public surfaces; #101–#103 closed as superseded by OBI's native dual-sided attribution; #90 webhook → v0.6).
• v0.5.1 Hardening landed on v0.5.1/phase-1-hardening (per ADR-0027): an adversarial multi-agent review of the merged v0.5 tree surfaced a cluster-wide :6443 custom-metrics auth bypass (CRITICAL) and six HIGH correctness bugs that green-CI self-merge had masked. All seven are fixed with regression tests: :6443 now requires the aggregation front-proxy client cert (new internal/frontproxy, requestheader CA pinned as ClientCAs); the read fan-out runs agents as secondary queriers with a mid-stream error swallow so one dead agent degrades instead of aborting; self-obs series carry a k8s_node_name label so per-node series stop merging; the custom-metrics adapter 404s on NaN/Inf instead of feeding the HPA a garbage int64; the forwarder widens its label schema to the union instead of freezing first-seen; and direction joins the #144 allowlist so L4 flow datapoints stop collapsing. The MEDIUM/LOW backlog (enumerated in ADR-0027) rides v0.6.
• v0.6 Hardening is next: TLS/CA everywhere the v0.5 posture deferred it (agent↔query hops, the custom-metrics APIService serving-cert CA bundle, the #90 validating webhook), HTTP/2 + gRPC capture modules, TLS uprobes, path templating, sampling, cardinality enforcement — plus the ADR-0027 MEDIUM/LOW backlog (HPA metricLabelSelector honoring, CEL per-span error isolation, drop-oldest span ring, TestIobsctl false-green, 400Mi agent memory limit, staleness markers, readiness gating on :9091).