home-ops AGENTS.md

PROJECT KNOWLEDGE BASE

Home-ops GitOps repo for a 3-node Talos Linux Kubernetes cluster managed by Flux v2. CLAUDE.md is a symlink to this file. For narrower operator detail see talos/AGENTS.md and bootstrap/AGENTS.md.

Core stack: Talos Linux (immutable OS) + Flux v2 (GitOps) + Cilium (CNI, BGP LoadBalancer, kube-proxy replacement, no L2 announcements) + Rook-Ceph (storage) + External Secrets Operator/1Password (secrets) + Cloudflare Tunnel (external access) + External-DNS (split DNS: network/cloudflare-dns public, network/unifi-dns internal) + VolSync (triple-destination backup). Gateway API HTTPRoutes front envoy-internal/envoy-external gateways in network. Monitoring: kube-prometheus-stack + Loki + Promtail + Tempo + Grafana + Alertmanager + Gatus + kromgo + KEDA + unpoller, all in monitoring.

STRUCTURE

.
├── kubernetes/
│   ├── apps/           # 19 namespaces at apps/base/<ns> + overlay apps/main/<ns> (see NOTES)
│   ├── clusters/main/  # Flux entry point: meta.yaml + apps.yaml (see NOTES)
│   └── components/     # alerts, common, dragonfly, volsync, kopiur (Stage 3 parallel)
├── talos/              # minijinja templates (see talos/AGENTS.md)
├── bootstrap/          # just bootstrap stages (see bootstrap/AGENTS.md)
├── .taskfiles/         # included: 1password, k8s, flux, rook, network, actions-runner
├── terraform/          # OpenTofu stacks for config outside Flux's reach (see NOTES)
├── docs/               # runbooks, incident history, ceph/network notes
└── .renovate/          # Renovate presets

Gatus is an app under kubernetes/apps/base/monitoring/gatus, not a component. Endpoints come from the gatus.home-operations.com/endpoint HTTPRoute annotation (auto-discovered) or hand-written entries in app/resources/config.yaml for non-HTTPRoute checks - not the dead gatus.io/enabled ConfigMap pattern (see NOTES).

WHERE TO LOOK

Task	Location	Notes
Add new app	kubernetes/apps/base/{namespace}/{app}/ + overlay kubernetes/apps/main/{namespace}/{app}.yaml	Overlay is a Flux Kustomization CR (one yaml per former ks.yaml). All 19 namespaces are on this layout. See NOTES.
Add app to namespace	kubernetes/apps/main/{namespace}/kustomization.yaml	Add - ./{app}.yaml
Enable backups	overlay kubernetes/apps/main/{ns}/{app}.yaml	VolSync: spec.components + dependsOn: volsync (namespace system) + VOLSYNC_* keys (example: media/seerr.yaml). Stage 3 dual-engine also adds components/kopiur + measured KOPIUR_* keys - owner: kubernetes/components/kopiur/Readme.md
App secrets	kubernetes/apps/base/{ns}/{app}/app/externalsecret.yaml	OnePassword via ClusterSecretStore onepassword
Bootstrap secrets	bootstrap/kustomize/apps/security/	vals injects ref+op://Home-Lab/1password/*
Flux entry point	kubernetes/clusters/main/{meta,apps}.yaml	cluster-meta -> cluster-apps dependency chain
Helm/OCI repos	kubernetes/apps/base/flux-system/meta/repos/	3 repo yaml files plus kustomization.yaml
Talos node config	talos/machineconfig.yaml.j2 + talos/nodes/*.yaml.j2 + talos/schematic.yaml.j2	Rendered by just talos, not Flux
Task commands	Taskfile.yaml + .taskfiles/{domain}/	task --list-all. Split with just: see UNIQUE STYLES
CI workflows	.github/workflows/	flate, renovate, codeql, image-pull, label-sync, validate, terraform-diff, terraform-publish, ai-pr-review, plus build-talosctl-busybox, labeler, tag, test-runner
Branch protection	GitHub ruleset on main, applied via gh api (not in Git)	docs/branch-protection.md. Only Labeler - Labeler is a required status check today - flate/image-pull/validate all path-filter at the on: pull_request: trigger level, so they never post a check outside their paths and are unsafe to require as-is (see doc for the live-measured proof and the follow-up needed to close the gap)
Renovate config	.renovaterc.json5 + .renovate/	Presets/config (extends Aviator-Coding/mortyops + local, incl. .renovate/talos.json5). Live writer is the in-cluster CronJob (kubernetes/apps/base/renovate/); GHA .github/workflows/renovate.yaml schedule is commented out as two-minute rollback. Owner: kubernetes/apps/base/renovate/README.md
Tool versions	.mise.toml	kubectl, flux, talos, helm, kustomize, vals, 1password-cli, just, minijinja, etc. Resolve with mise which <cli> / mise exec (see NOTES).
Authentik SSO config	terraform/authentik/ (OpenTofu)	Adopted apps/providers via import blocks, plus stack-created LiteLLM OIDC (litellm.tofu). Other flows/stages/mappings stay Authentik-blueprint data sources. Never tofu apply without an explicit go-ahead. Runbook: docs/authentik/terraform.md
AI stack	kubernetes/apps/main/ai/ (Flux Kustomizations) + kubernetes/apps/base/ai/ (manifests)	Hermes + ToolHive (toolhive.stacklok.dev/v1alpha1 MCPServer) + agentgateway + LiteLLM (governance + fallback chains; internal route only since 2026-08-26, never the public listener; delivered by litellm.home-operations.com/v1alpha1 CRs from ai/litellm-operator - docs/ai-system/litellm/{README,fallbacks}.md). kagent/kmcp tombstones: docs/ai-system/{kagent,kmcp}. Retired 2026-08-22: docs/ai-system/retired-2026-08-22.md
konflate (PR review UI)	kubernetes/apps/base/flux-system/konflate/	Read-only Flux PR-review UI, internal HTTPRoute only. Write-back is off and no GitHub credential is in-cluster (public-repo anonymous reads). Do not copy the reference repo's write-back / shared GitHub App wiring.
PVC write-access check	kubernetes/apps/base/system/pvc-writable-check/	CronJob (every 6h) execs test -w <mountPath> in every PVC-mounting container and alerts via PrometheusRule on genuine failure - the check that would have caught the autobrr/rsshub-playwright empty-volume bugs (2026-08-30). Cluster-wide pods get/list plus per-namespace pods/exec: create RoleBindings that deliberately omit rook-ceph, database, and security (API-server denial, not script-only); design rationale and that RBAC tradeoff are in the app's README. Prefer volumeMounts[].readOnly: true for expected read-only (headlamp); the pod-wide skip annotation is an unused escape hatch.