fallback-required

Fallback Required

The Law

LLM CALLS WITHOUT FALLBACKS ARE TICKING FAILURES.
Every model times out. Every model returns garbage sometimes.
"The model is reliable" is a claim about averages — users experience tails.
A defined, tested fallback path for each failure mode IS reliability.

When to Use

Trigger on every PR that:

• Adds a new LLM API call
• Changes existing error handling on a model call
• Adds streaming or async generation
• Introduces tool use or agentic function calls

When NOT to Use

• Offline batch jobs where failures can be retried with human review and no user is waiting
• Exploratory prototypes not going to production

The Four Failure Modes

Every LLM call must handle all four:

Failure Mode	What Happens	Required Response
Timeout / API error	Network down, provider outage, slow response	Retry with exponential backoff (max 3), then graceful degradation
Malformed output	Wrong format, truncated JSON, schema violation	Schema validation → fallback to rule-based default
Low confidence	Model expresses uncertainty, output score below threshold	Route to fallback model, simpler rule, or human review
Refusal	Model declines to answer, content filter triggered	Detect refusal pattern → user-friendly error, do not surface raw refusal

The Process

Step 1 — Define the Fallback Before Writing the Call

Before writing the LLM call, answer: what does this feature return when the model fails?

The fallback must be:

• User-safe — no error stack traces, no raw model output
• Defined — not "we'll figure it out" but a concrete response or behaviour
• Logged — every fallback invocation records the reason

Step 2 — Implement All Four Handlers

async def call_llm(prompt: str) -> Result:
    for attempt in range(MAX_RETRIES):
        try:
            response = await llm.complete(
                prompt, timeout=TIMEOUT_SECONDS
            )
            parsed = parse_and_validate(response)   # raises OutputParseError on bad schema
            if parsed.confidence < CONFIDENCE_THRESHOLD:  # default 0.7; use 0.85 for high-stakes domains
                log_fallback("low_confidence", attempt)
                return fallback_result(reason="low_confidence")
            return parsed
        except TimeoutError:
            if attempt == MAX_RETRIES - 1:
                log_fallback("timeout", attempt)
                return fallback_result(reason="timeout")
            await backoff(attempt)
        except OutputParseError:
            log_fallback("malformed_output", attempt)
            return fallback_result(reason="malformed_output")
        except RefusalError:
            log_fallback("refused", attempt)
            return fallback_result(reason="refused")
    return fallback_result(reason="max_retries_exceeded")