harm-anticipation

Harm Anticipation

Harm anticipation is systematically thinking through how an AI product could cause harm — before it does. Preventive design, not reactive crisis management.

The work is unglamorous and easy to skip. Done well, it produces specific testable mitigations. Done badly, it produces a doc nobody reads.

Categories of AI harm

• Direct harm: the AI outputs something harmful — dangerous advice, discriminatory content, privacy violations
• Facilitated harm: the AI helps a user do something harmful, even if the AI's output itself is benign
• Emergent harm: harmful patterns from scale or interaction effects, not from any single output
• Omission harm: the AI fails to act when it should — not flagging a crisis, not escalating
• Erosion harm: gradual negative effects — dependency, deskilling, manipulation, trust erosion

Structured anticipation

Work through each harm category with five lenses:

1. Who could be harmed? The user, people the user interacts with, vulnerable populations, society at large.
2. How could they be harmed? Physical, emotional, financial, reputational, privacy, autonomy.
3. What's the likelihood? Common use case vs. edge case vs. adversarial attack.
4. What's the severity? Inconvenience vs. distress vs. irreversible damage.
5. What's the detectability? Obvious and immediate vs. subtle and delayed.

Misuse scenarios

Think like an adversary:

• How would someone deliberately misuse this feature?
• What's the easiest way to extract harmful output?
• Could this be used to manipulate, deceive, or coerce?
• What if the user lies about their intent?
• What happens at scale, automated?

Unintended consequences

Think second-order:

• What happens at millions of users?
• What skills do users lose by relying on this?
• Could this create unfair advantages or disadvantages?
• What power dynamics shift?
• What becomes possible that wasn't before — for better and worse?

Decision rules

• Score by frequency × severity, not severity alone. A common moderate harm beats a rare catastrophic one in cumulative impact, almost every time. Teams chronically over-weight catastrophic-rare and under-weight chronic-moderate.
• Each harm needs a falsifiable test. "If the AI does X to user type Y, the system should Z." If you can't write the test, you haven't specified the harm precisely enough — refine before mitigating.
• Mitigate at the lowest layer that works. Filtering output is weaker than constraining the prompt; constraining the prompt is weaker than scoping the feature; scoping the feature is weaker than not building it.
• Re-anticipate after each scale jump. Harms at 1k users differ from 100k differ from 10M. The same checklist gives different answers at each tier.
• Vulnerable users define the floor, not the ceiling. If the worst-off user is well-served, others will be too. Designing for the median user with edge-case patches is the wrong shape.
• Prefer reversible mitigations during the discovery phase. Lock-in early and you'll be scared to revisit assumptions.