satellite-observability

Observing production satellites

Facts verified 2026-10-09. Observability infra is being migrated; if a datasource below fails, run list_datasources once and say what changed in your report.

What exists (and what does not)

Signal	Status	Where
App metrics (monkit)	Yes, all prod regions	VictoriaMetrics ffqrvx0pyhp8gf (Grafana default)
k8s pod metrics	Yes	same datasource
Prod satellite logs	Yes (most peers)	ClickHouse otel.otel_logs via cfqvdb4b1y96oa — see Logs below
Traces / profiles	No satellite data in Tempo/Pyroscope	live pprof via satellite-debug-port skill
Eventkit events	Not in Grafana	ClickHouse warehouse (if you have access)
Prod config, version	git	satellite-infra skill
Live process state	debug port: /mon/stats, /debug/pprof, /metrics	satellite-debug-port skill

Logs (ClickHouse otel.otel_logs)

Prod satellite logs are in ClickHouse, table otel.otel_logs, through the Grafana datasource cfqvdb4b1y96oa (grafana-clickhouse-datasource). They are not in VictoriaLogs. Human view: dashboard otel-logs-explorer (https://grafana.storj.tools/d/otel-logs-explorer). Verified 2026-10-09: all four regions, near real time, about 9 days of retention (oldest rows on 2026-10-09 were from 2026-09-30).

Run SQL through the Grafana API (read-only SELECT):

grafana_api_request POST /api/ds/query
{"from":"now-1h","to":"now","queries":[{"refId":"A","format":1,
  "datasource":{"uid":"cfqvdb4b1y96oa","type":"grafana-clickhouse-datasource"},
  "rawSql":"SELECT ..."}]}
jq: .results.A.frames[0].data.values

Tip: build one string column with concat(...) and read .values[0] — easier than many columns.

Columns that matter:

• Timestamp — always filter on it first (huge table: us1 alone is millions of lines per hour).
• ResourceAttributes['environment_name'] — storj-prod-satellite-us1 etc.
• `__otel_materialized_k8s.namespace.name`, `__otel_materialized_k8s.pod.name`, `__otel_materialized_k8s.container.name` (= satellite for the app container). Use these materialized columns, not the map, for speed. Pod name = <app>-<hash>-<id>, so filter a peer with pod.name LIKE 'satellite-api-%'.
• ResourceAttributes['server_name'] — the host (bare-metal repairers have no k8s fields).
• Body — the line. SeverityText, ServiceName, ScopeName are empty for satellite logs.