layered-logging-and-alerting-patterns

Layered Logging and Alerting

In some services, a single logger.Error(...) call is not "just a log line" — it can simultaneously capture an event to an error tracker, ship the log to a centralized collection endpoint, and (for certain severities or subsystems) fire a chat/webhook alert. Treating a call site as if it only writes local output is the source of most surprising regressions in this shape of system — the fix here is understanding what a given log call actually fans out to before changing or removing it.

A log call can be a multi-sink side effect

func Errorf(ctx context.Context, format string, args ...any) {
    msg := fmt.Sprintf(format, args...)
    logToLocalOutput(msg)
    captureToErrorTracker(ctx, msg)        // e.g. Sentry
    go sendToCentralizedLogEndpoint(msg)   // async, fire-and-forget by design
}

Before removing or "simplifying" a logging call in a codebase with this shape, check what it actually does — an apparently-redundant log line can be the only thing populating an external dashboard or triggering an on-call alert; deleting it silently removes that signal with no local indication anything changed.

Async side effects need a flush guarantee on exit paths

If a log call ships data asynchronously (a background goroutine posting to an external endpoint), a process that exits immediately after logging a fatal error can terminate before that goroutine's request completes, silently losing the log. A deliberate short delay before actually terminating (after a Fatal/Panic-style log call specifically) is a real, intentional pattern to give an in-flight async log delivery time to finish — not dead code or an accidental performance issue to "optimize away."

Sampling and filtering are usually deliberate, not accidental

Error-tracking integrations commonly sample by environment (log everything in development, sample a smaller percentage in production to control volume/cost) and filter certain error classes before sending (a known- noisy, low-value error message excluded via a beforeSend-style hook). Both are typically deliberate operational decisions — verify the actual current sampling rate and any filter rules before assuming an error "isn't being tracked" is itself a bug, versus a documented filtering choice.