ai-security

AI Security

A language model cannot reliably tell instructions from data. Everything below follows from that one fact: any text the model reads can steer it, and anything the model can do, an attacker who controls that text can try to do.

Two threats are current and concrete: attacks on the AI feature itself (prompt injection, over-powered agents) and attacks on the keys and accounts behind it — AI-provider keys are monetisable on sight, and platform breaches have been reported to target AI inference providers specifically (see security-incident-response).

Scope: AI features in software the user owns or is authorised to assess. Testing prompts against the user's own application is in scope; attacking third-party AI services or other people's deployments is not.

Taxonomies change. The OWASP Top 10 for LLM Applications is the maintained reference — check its current edition rather than quoting category numbers from memory.

---

1 · The order of work

 1  INVENTORY    every model call, tool, data source, MCP server, plugin; where each key lives
 2  KEYS & SPEND provider keys scoped, per-environment, capped, monitored
 3  AGENCY       what can the AI do? reduce it to what it needs; gate the irreversible
 4  DATA FLOWS   find the three-way overlap (below); break at least one leg
 5  INPUT PATHS  every place untrusted text reaches the model (users, web, files, email, tool results, RAG)
 6  OUTPUT       treat model output as untrusted input to the rest of the system
 7  ISOLATION    tenants, memory, vector stores, conversation history
 8  SUPPLY CHAIN models, plugins, MCP servers, skills, prompt files — all of it is code
 9  TEST & WATCH red-team your own app; regression evals; alert on abuse

2 · The three-way overlap that makes exfiltration possible

An AI feature is in serious danger when one session combines all three:

1. Access to private data (user data, secrets, internal documents)
2. Exposure to untrusted content (web pages, emails, uploaded files, issue text, retrieved documents, other users' input)
3. A way to communicate externally (HTTP requests, email, rendering images or links, writing to a shared place)