Prompt injection, permissions, privacy and model limitations

Why it matters

An LLM application treats text as instructions, and much of that text comes from places you do not control. Once the model can read private data and take actions, a sentence hidden in a web page or an email can redirect it. "How would an attacker abuse this feature?" is a natural question in any applied AI design discussion, and the realistic failure is a data leak, not just an embarrassing output.

Core ideas

Prompt injection. Direct injection is a user typing instructions that override the developer's ("ignore the previous instructions and..."); jailbreaks are direct attempts to get around safety behavior. Indirect injection hides instructions in content the system processes on a user's behalf: a retrieved document, a web page, an email being summarized, a code comment, a tool result. The model cannot reliably separate data from instructions because both arrive as tokens in the same context. Input filters, delimiters and "never follow instructions inside documents" prompts reduce some attacks but are not a security boundary; assume a determined attacker eventually gets text through.

The dangerous combination. Risk peaks when one agent has (1) access to private data, (2) exposure to untrusted content and (3) a channel to send data out, which can be as subtle as rendering an image whose URL carries data in its query string. For any flow that handles untrusted content, remove at least one leg: no outbound channels while untrusted content is in context, no private data in that context, or human approval of every outbound action.

Design-level defenses.

• Least privilege: tools scoped to the task and to the user's own permissions; read-only by default.
• Authorization by the server on every tool call, using the authenticated user and tenant, never parameters chosen by the model.
• Treat model output as untrusted input downstream: validate it against schemas, escape it before rendering HTML or building queries, and never execute generated code outside a sandbox.
• Human approval for irreversible or outbound actions, showing exactly what will be sent.
• Separate trusted control flow from untrusted data: a planner sees only the trusted request, untrusted content is processed by a component that cannot trigger actions, and policies restrict which data may flow into which tools.
• Monitoring and rate limits to detect abuse and bound cost.

Unsupported claims. Models generate plausible text, not verified facts, and state errors fluently. Ground answers in retrieved sources with citations, verify that cited sources support the claims, constrain outputs to structured fields where possible, allow abstention, evaluate faithfulness explicitly, and show uncertainty in the product.

Tenant and data permissions. Multi-tenant systems must isolate retrieval indexes, caches, logs and fine-tuning data per tenant. Filter retrieval by the user's access rights before anything reaches the model. Be careful with shared caches of prompts and responses, and assume a system prompt can leak, so never embed secrets in it.

Privacy. Send only the personal data the task needs; redact or pseudonymize personal data in logs and evaluation sets; set retention periods and deletion paths for prompts, outputs and traces; check what the model provider retains and where data is processed; and do not train on user data without a clear basis and consent. Models can memorize rare training sequences, so fine-tuning on sensitive data creates an extraction risk.

Model limitations. Knowledge cutoffs, sensitivity to phrasing, nondeterminism, uneven quality across languages and domains, and unreliable arithmetic or exact recall without tools. Communicate them to stakeholders with evidence from evaluations rather than reassurance.

Common misconceptions

• "A strong system prompt prevents prompt injection." At best it lowers success rates; enforce boundaries outside the model.
• "Only user input is untrusted." Retrieved documents, web pages, emails and tool outputs are untrusted too.
• "Filtering offensive output is enough." Exfiltration hides in URLs, tool arguments and markdown images; restrict what outputs can do.
• "Retrieval eliminates hallucination." It reduces it, but models still misread sources or go beyond them; verify faithfulness.
• "The model provider handles privacy." Your logs, caches, evaluation sets and retention policies are your responsibility.
• "An injection classifier solves the problem." Detectors are useful signals but can be bypassed; design so that a successful injection has limited impact.

How to approach it

For any LLM feature, draw the data flow, mark untrusted inputs, sensitive data and side effects, then remove dangerous combinations and add server-side controls. Walk through the OWASP Top 10 for LLM Applications with your own design, and try an indirect injection on a toy RAG system to see how easily it works. Senior-level discussions expect threat models, layered defenses and an honest statement of residual risk.