Dev.to · 2 min read

Threat modeling LLM tool-calling

Threat modeling LLM tool-calling

Every field above is part of the contract enforced by publisher validate. This post exists so the pipeline always has something real to plan against. Why tool-calling changes the threat model A language model that only emits text has one output channel: the reply. A model that can call tools has as many output channels as it has tools, and each of them is reachable by anything that can get text into the context window — a fetched page, a code comment, a file name, a CI log. The useful framing is that the context window is an untrusted input, and every tool is a sink. Prompt injection is not a new vulnerability class so much as a confused-deputy problem wearing a new hat. Three questions per tool For each tool exposed to a model, write down: What does it read? A tool that reads secrets turns any injection into an exfiltration primitive, whether or not the model "intends" it. What does it change? Distinguish reversible writes from irreversible ones. Deleting a branch and deleting a production table sit in different buckets. Who authorised it? Authority should ride with the request, not with the agent process. An agent running as a service account is an agent with the union of every user's permissions. A minimal mitigation set Scope credentials per invocation, not per deployment. Make irreversible tools require an out-of-band confirmation that the model cannot itself produce. Log the full tool-call payload, not a summary. The summary is written by the thing you are investigating. None of this is exotic. It is ordinary least-privilege design applied to a component that happens to take instructions from strangers.

This is a summary aggregated from Dev.to. Read the complete article on the original site:

Read full article at Dev.to

More AI & Machine Learning News