One approval, two payments: what an AI agent does after a crash

Written with help from AI (Claude). I maintain Call4Agents, the library mentioned at the end. Every number here comes from scripts you can rerun, and the fixes in the "What to do" list work without my library. A person approves one payment. The tool pays. The process dies before the framework saves its progress. A new process resumes the way the docs say. The payment runs again. I measured this in LangGraph, Pydantic AI and the OpenAI Agents SDK. In all three, one approval gave 2 payments, in every run: 30 runs per framework between 21 October and 5 November, on the newest versions each day. A guard that records the action outside the framework brought it to 1 payment, every run. This isn't a framework bug. It's at-least-once execution, which is what any system gives you when it saves progress after doing the work. LangGraph's docs already warn about it. OpenAI added a docs section after I asked. Pydantic AI's triage reproduced it. The trouble is the gap between what "approved" sounds like and what it promises. People read "a human approved this" as "this happens once, exactly as approved". Neither part is guaranteed. The experiment Fake models, no network, no API keys. Each step is a separate OS process. The crash is os._exit, so no cleanup code runs. A "payment" is a line added to a local file. Steps: the agent asks for pay(amount=10, to="alice"), which needs approval. A person approves, the tool pays, and the process dies right after. A new process resumes from the saved state. Count the payments. A second case: the saved call is changed to pay(9999, "mallory") after the pause and before it runs. That models anyone who can write to the state store. Framework Crash after the approved payment Saved call edited after the yes LangGraph 2 payments paid 9999 to mallory Pydantic AI 2 payments paid 9999 to mallory OpenAI Agents SDK 2 payments refused by the SDK itself Any of them, with a guard outside 1 payment nothing paid Two details I don't want to hide: The OpenAI SDK deserves credit. It refuses changed arguments under the same call id on its own. LangGraph sometimes asks for approval again, and I can't predict when. When it asks, the second payment follows a second "yes" from someone who may not know the first one went through. It paid twice either way. Try it in five seconds pip install call4agents call4agents incident This replays three failures, without and with the guard. My run on 6 November took 5.5 seconds: Incident 1. The double payment A person approves pay(10, alice). The bank charges. The process dies before saving. It resumes. without Call4Agents charges: 2 paid: 10 to alice, 10 to alice with Call4Agents charges: 1 paid: 10 to alice The same, but the process dies inside the call, so nobody knows if it finished: without Call4Agents charges: 2 paid: 10 to alice, 10 to alice with Call4Agents charges: 1 paid: 10 to alice (stopped: in doubt, a person must check) Incident 2. The changed call A person approves pay(10, alice). The saved call is changed to pay(9999, mallory) before it runs. without Call4Agents charges: 1 paid: 9999 to mallory with Call4Agents charges: 0 paid: nothing (refused: no yes for this exact call) Incident 3. The ignored no A person says no to pay(10, alice). A bug in the agent loop runs the call anyway. without Call4Agents charges: 1 paid: 10 to alice with Call4Agents charges: 0 paid: nothing (refused: no yes for this exact call) The processes and crashes are real. The bank and the agent loop are small local stand-ins, not the frameworks. Look at the second line of Incident 1. When the process dies inside the call, nobody knows whether the payment went through. The guard doesn't guess and doesn't run it again: it stops as "in doubt". Then a provider check, an idempotency key or a person decides. I think that's the honest answer, because no library can know what happened in someone else's bank. Why this happens Doing the work and saving that you did it are two separate steps, and no process can make them one. If the crash falls between them, the saved state still says "not done", so a resume does it again. Approval adds another trap. Approval state is usually a flag next to the call: this call id was approved. Nothing ties the "yes" to the exact arguments, to one use, or to a real pause. My own bug The library had this kind of bug too. A reader found that in every release from 0.1.0 to 0.5.0, a "yes" for 10 could let 20 run, with only a crash at the wrong moment. All my checks passed the whole time. It's fixed in 0.5.2, with a public advisory. After that I added a test that kills a real process before and after every write, 412 crashes in total. With the old bug put back in, that test failed by itself. It would have found my bug before the reader did. What to do, with or without my library Use an idempotency key the provider enforces, stable across resumes and scoped to your operation. Treat "maybe happened" as unknown. Without a provider key, don't replay automatically. A read that finds nothing can lag. Check the approval at the tool, against the arguments about to run, not only at the resume. Crash your own agent on purpose. Kill the process between the action and the save. It takes minutes. Limits Fake models, one machine, no real payment provider, no production traffic. Three frameworks at a few versions, with the crash at one point. It's a small experiment you can repeat, not a production study. The full write-up, with the script and every run's output, is here: https://github.com/salahmakboul/call4agents/tree/main/docs/finding If you run an approval flow, try to break it the same way and tell me what happens, with or without my library.