pod-preemption

Pod Preemption

The tell: failed far exceeds the number of failed indexes

An indexed Job reports both counts. Read them together:

kubectl -n geo-workflows get job <name> \
  -o jsonpath='succeeded={.status.succeeded} failed={.status.failed} failedIndexes={.status.failedIndexes}{"\n"}'

A real example from the CHELSA build (data-workflows #564):

succeeded=120 failed=47 failedIndexes=39,43

122 indices, only two ultimately failed — so roughly 45 pods died and were retried successfully. That ~28% pod mortality is the signal. failed counts pod deaths, not indices; when it is many times larger than len(failedIndexes), the work is fine and the pods are being killed underneath it.

A job can even report Complete=True while this is happening. Retries hide it. Check the gap on every long job, not just failing ones.

Why it happens

priorityClassName: opportunistic on NRP carries priority -2000000000 — the lowest value available. Verify what a pod actually got:

kubectl -n geo-workflows get pods -l <selector> \
  -o custom-columns='NAME:.metadata.name,PRIOCLASS:.spec.priorityClassName,PRIO:.spec.priority'

Preemption risk scales with how long a pod runs. A 5-minute opportunistic pod is nearly always fine; a multi-hour one is a large target and will be evicted repeatedly. Pod runtime is the variable that matters, far more than pod count.

Do not misread it as a data problem

The failing indices are usually not special. In #564 the two that failed looked meaningful — both were sparse-land ocean cells — and that coincidence cost two diagnostic cycles. The same h0 then:

run	retries per index	outcome
parallelism 40	3 attempts	failed
parallelism 2	2 attempts	failed
single pod	1 attempt	succeeded, unchanged

The variable was the retry budget against a background death rate — not the cell, not the code, not parallelism. Before investigating the data, re-run one failing index on its own. If it succeeds unchanged, it was environmental.