home-ops: Skill for Claude Code

Rook-Ceph: cluster config, CephX, RGW and OSD health

Rook is the only writer of Ceph config. A green ceph status does not mean RGW writes work, and a toolbox ceph config set survives as mon-DB drift that Git will not reconcile away.

Tripwires

1. GitOps only. Change tunables in cluster/helmrelease.yaml cephConfig. Never ceph config set or ceph auth by hand. Before trusting a number, diff ceph config dump against that file. Three live keys are mon-DB drift, not Git: osd_max_backfills=1, osd_mclock_override_recovery_settings=true, osd_recovery_sleep_ssd=0.05. A destroyed OSD can also leave a who=osd.N row; ceph config get on a missing id returns the built-in default and looks like a stored key. Use the dump's who column. baseline.md
2. rgw_sigv4_insecure: "true" stays until the image bump that removes it. Ceph v20.2.4 rejects minio-go PUTs (unsigned content-type) while GET/HEAD still succeed, so HEALTH_OK does not clear RGW. Clear the flag only in the same change as a Tentacle image that contains ceph/ceph#71192. quay.io/ceph/ceph has no never-automerge rule: a stable x.2.z patch still matches the generic patch automerge and can merge a green PR while leaving the flag set. decisions.md
3. security.cephx.csi stays aes. Kernel clients need Linux >= 7.0; these nodes run 6.18.x. Only daemon is on aes256k. keyGeneration only increases. A CRD older than Rook 1.20.5 prunes keyType and burns a generation on an aes rotation.
4. New RWX claims use ceph-filesystem-rwx (group csi-rwx). The default csi subvolumegroup still returns EINVAL on create/getpath. Do not delete the csi-rwx CR because the image is past v20.2.2.
5. Judge reclaim with rbd du --exact. Plain rbd du counts 4 MiB objects that still exist after 64 KiB discards. mountOptions: [discard] on ceph-block stays off. Weekly fstrim reaches mounted claims only.
6. osdMaxUpdatesInParallel: 1 lives under spec.storage. The CRD default is 20. The spec-level key is pruned and does nothing. Never restart all OSDs. Before a node reboot: HEALTH_OK and task rook:check-osd-device-paths (Rook #17224). Skill talos-nodes.
7. Do not delete csi-driver-rbac.yaml or csi-driver-tolerations.yaml. Both are partial manifests for the out-of-band ceph-csi-drivers Helm release. The tolerations one sets prune: disabled; SSA merges spec.nodePlugin.tolerations only.
8. OSD memory request stays 12Gi, limit 14Gi. osd_memory_target (10 GiB) sizes the BlueStore cache, not RSS. The highest observed OSD working set is 9317 Mi. 10 Gi is not a free cut. Mon and logcollector stay Guaranteed (requests == limits). An empty limits: {} does not clear the operator chart's 512Mi default; the operator limit is an explicit 1Gi.
9. rook and nfs mgr modules stay off. The rook module crash-looped and blocked node upgrades. nfs is unused and was evaluated with the discard work.
10. StorageClass parameters are immutable. A change is a new class, not an edit. No deviceClass on pool data0. MDS anti-affinity stays preferred, not required. Ceph alert counters go through increase(), never a raw > N compare.
11. The metadata CronJob is not last-resort DR. ceph-backup-pvc is OpenEBS hostpath on this cluster. A host wipe destroys it. Steps: backup/RECOVERY-PROCEDURES.md.