securing-ai-systems

Securing AI Systems

LLM applications break the assumption every other security control is built on: that instructions and data are separable. In an LLM, data is instructions. Every design that reads untrusted content and then acts has to be evaluated with that in mind, and no amount of prompt engineering fixes it.

When to Use

• Security review of an LLM-backed feature, chatbot, or copilot
• Threat modeling an agentic system: tools, autonomy, memory, multi-agent
• Reviewing an MCP server, tool definition, or plugin surface
• Assessing a RAG pipeline for data leakage and poisoning
• Evaluating model, dataset, and dependency supply chain
• Red teaming an AI system with authorization

When NOT to Use

• Conventional web/API vulnerabilities in the surrounding app — use auditing-code-for-vulnerabilities, testing-web-applications, testing-apis. Most real AI-app breaches are still ordinary IDOR and SSRF.
• Building jailbreaks or attacks against third-party models you do not own or have authorization to test — out of scope
• Model safety alignment research — different discipline

Route to a Depth Skill

The MCP review here is one part of a wider AI threat model; reach for auditing-mcp-servers when the server implementation itself is the target.

The Core Rule

Treat every model output as untrusted user input, and every input the model reads as potentially adversarial instructions.

From that single rule, most of the correct architecture follows: never route model output into a sink without the same validation you would apply to a form field, and never grant the model an authority the least trusted content it will read should not have.

The Lethal Trifecta

An agent is exposed to serious compromise when it has all three of:

1. Access to private data (files, DB, internal APIs, user context)
2. Exposure to untrusted content (web pages, email, tickets, PRs, docs)
3. A way to communicate externally (HTTP, email, writes to a shared surface)