aws-cost-investigation

AWS Cost Investigation

Operational skill for diagnosing AWS cost spikes and auditing for ongoing waste. The focus is on diagnostic flow (data-first, not guess-first) plus a concrete trap inventory with detection CLI for each.

When to invoke

Symptoms:

• "The bill is up $X with no deploys" / "AWS bill spiked last month."
• An anomaly notification from AWS Cost Anomaly Detection.
• Cost Explorer dashboards show large (no tag) slices despite tagging policies.
• NAT Gateway charges growing month over month.
• Looking at a Savings Plan / Reserved Instance commitment decision.
• A general account audit ("find the waste").
• Designing a cost-allocation tagging strategy.

Cross-cutting rules

1. Data first, guesses never. When asked to diagnose a spike, the first action is to query Cost Explorer. Do NOT guess "probably S3" or "probably NAT" without numbers. Naming a likely culprit without data is anti-pattern #1.
2. Compare windows of equal length. A 7-day spike compares to the prior 7 days, not month-to-date. A monthly spike compares to the same days of the prior month, not the full prior month.
3. Never quote a specific dollar amount as a pricing fact. AWS prices change. State relative magnitudes (Gateway endpoints are free; Interface endpoints are cheaper than NAT for high egress) and link to the AWS pricing page for current numbers when a precise answer is needed.
4. Activation is the silent gate for tag-based analysis. A tag on a resource is invisible to Cost Explorer / CUR until it's activated as a Cost Allocation Tag. See Cost allocation tagging.
5. Stop-the-bleeding before re-architecting. When a runaway cost is identified, the first action is to cap it (set retention, delete idle, add a budget alarm) — not redesign the workload.

Diagnostic flow for spikes

Run the steps in order. Don't skip ahead.

Step 1 — Service-level diff

# Compare two equal-length windows (here: last 7d vs prior 7d)
END=$(date -u +%Y-%m-%d)
START=$(date -u -d '-7 days' +%Y-%m-%d)
PRIOR_END=$START
PRIOR_START=$(date -u -d '-14 days' +%Y-%m-%d)

for window in "$PRIOR_START $PRIOR_END" "$START $END"; do
  read s e <<<"$window"
  aws ce get-cost-and-usage \
    --time-period Start=$s,End=$e \
    --granularity DAILY \
    --metrics UnblendedCost \
    --group-by Type=DIMENSION,Key=SERVICE \
    --query 'ResultsByTime[].Groups[]' \
    --output json > /tmp/ce-$s.json
done
# Diff per-service totals between the two files to find the top growers.