Start from what the model can touch
An LLM feature is rarely just a model. It is a prompt template, a retrieval layer, a set of tools, and the data and permissions those tools carry. The useful security question is not whether the model can be tricked. Assume it can. Ask what happens when it is.
Draw the data flow on one page. List every source that can put text into the context window: users, uploaded files, retrieved documents, web pages, emails, tool results. Then list everything the model's output can cause: a reply, a database query, an API call, a ticket, a code change. Risk sits where untrusted input meets a consequential action.
The risks that matter in practice
The OWASP Top 10 for LLM Applications (2025 edition) is a good shared vocabulary. The entries that show up most in real assessments:
- Prompt injection: instructions in a user message, or hidden in content the model reads (indirect injection), that override what the developer intended.
- Sensitive information disclosure: the model repeats data from its context, training or retrieval that the current user should not see.
- Improper output handling: model output passed straight into a shell, SQL query, HTML page or another system without validation.
- Excessive agency: tools that hold more permissions than the task needs, or act without a human check.
- System prompt leakage: secrets or business rules placed in the system prompt, which should be treated as public.
- Vector and embedding weaknesses: retrieval that ignores the requesting user's permissions, so search becomes a data-leak path.
- Unbounded consumption: no limits on tokens, calls or cost per user, which turns into a billing or availability problem.
Controls that hold up
- Least privilege for tools: give each tool credentials scoped to the requesting user, not a shared service account.
- Human approval for high-impact actions: moving money, deleting data, changing access or sending messages outside the company.
- Authorization in the retrieval layer: filter documents by the caller's permissions before they reach the prompt.
- Output treated as untrusted input: validate against a schema, encode for the destination, use parameterized queries.
- No secrets in prompts: keys and tokens live in the tool layer, never in text the model can repeat.
- Provenance labels: mark which context is trusted instruction and which is untrusted content. This reduces injection; it does not remove it.
- Limits: per-user and per-tenant rate, token and spend caps.
- Logging: prompts, retrieved documents, tool calls and outputs, kept under the same retention rules as the data they contain.
Test it like an attacker would
Generic jailbreak scanners catch known strings. They do not understand your business logic. Test the specific workflow: plant instructions in a document the assistant will summarize, ask it to act on another tenant's record, try to make it leak data through a rendered link or image.
Retest after every model, prompt or tool change, because behavior shifts between versions. MITRE ATLAS catalogs adversary techniques against AI systems and works well as a scoping checklist.
Governance without paperwork
Keep an inventory of the models, providers, prompts and tools you run in production, each with an owner. If you need a structure to show auditors or the board, the NIST AI Risk Management Framework (Govern, Map, Measure, Manage) and ISO/IEC 42001 both fit. Start with the inventory and the feature with the most agency.