observability

Observability Skill (Nebari + Generic Kubernetes)

Use this skill whenever the user asks to:

• debug deployments using logs/events/rollout state,
• verify monitoring is working,
• find Grafana/Loki/Prometheus/Mimir/Tempo endpoints,
• check whether Pi users can access observability data.

This skill supports:

• local k3s/k3d,
• cloud Kubernetes (EKS/GKE/AKS),
• generic Kubernetes environments.

---

Core design awareness (important)

In Nebari Pi environments, the common architecture is:

• Pi/JupyterHub workloads in an app namespace (often data-science),
• LGTM stack in monitoring,
• optional pi-observability-proxy and pi-debug CLI for curated debugging.

Access paths you must evaluate

1. pi-debug path (if available):
   • pi-debug logs|events|rollout|doctor
   • Usually fronted by pi-observability-proxy.
2. Direct in-cluster LGTM path:
   • Loki/Mimir/Tempo/Grafana service DNS from within Pi pod.
3. Port-forward/browser path:
   • useful when running from local workstation or when pod egress is restricted.

Do not assume only one path exists. Detect and use what works.

---

Workflow

0) Detect environment + tools

Run:

kubectl config current-context
kubectl get ns

Then detect whether pi-debug exists:

command -v pi-debug >/dev/null && echo "pi-debug available" || echo "pi-debug not available"

If pi-debug is available, test quickly:

pi-debug doctor --app pi || true

---

1) Discover LGTM endpoints (do not hardcode)

Preferred discovery:

kubectl get svc -A | grep -Ei 'grafana|loki|mimir|tempo|prometheus'

Export discovered endpoints (examples):

export GRAFANA_SVC="http://<grafana-svc>.<ns>.svc.cluster.local:80"
export LOKI_URL="http://<loki-svc>.<ns>.svc.cluster.local:3100"
export MIMIR_URL="http://<mimir-gateway-svc>.<ns>.svc.cluster.local"
export TEMPO_URL="http://<tempo-svc>.<ns>.svc.cluster.local:3200"
export PROM_URL="http://<prometheus-svc>.<ns>.svc.cluster.local:9090"