Retries Are Not Free: Five Myths, an Attempt Ledger, and a Stop Rule

A job failed at 2am. It passed on the third try. Everyone went back to sleep. Two hours later, support found three identical orders. The retry worked. The side effects did not. So what does for attempt in range(3) actually promise you? Nothing. It is a loop, not a contract. This is the FAQ I keep needing when someone says "just add retries." Free model access and a free server make retries feel cheap. They are not cheap. They are invisible. Myth 1: "A retry is the same request, sent again" Test: send one write twice. Count the rows. A retry is a new request with the same intent. The wire does not know your intent. Only you do. Give the intent a name. Hash the payload. Store the hash next to the effect. import hashlib, json def idem_key(payload: dict) -> str: blob = json.dumps(payload, sort_keys=True).encode() return hashlib.sha256(blob).hexdigest()[:32] Corrected model: a retry is safe only when the receiver can recognize a replay. Most model APIs will not dedupe for you. Your own store has to. Myth 2: "Exponential backoff is polite" Backoff without jitter is a synchronized herd. Twenty workers fail together. Twenty workers retry together. Test: log attempt timestamps from 20 workers. Look at the gaps. They are identical. Use full jitter. Pick a random point inside the window, not the window's edge. sleep = min(cap, base * 2 ** (attempt - 1)) time.sleep(random.uniform(0, sleep)) The AWS Architecture Blog write-up on backoff and jitter is still the clearest reference. Read it before you copy a snippet from a gist. Corrected model: backoff controls your rate. Jitter controls correlation. You need both. Myth 3: "More retries means more reliability" Count the multiplier. Three layers with four attempts each. One user click becomes 64 calls. That is retry amplification. Google's SRE book covers the pattern in Addressing Cascading Failures. A struggling dependency gets hammered hardest exactly when it can least take it. Set a stop rule with three parts: a deadline, in wall-clock seconds; a hard attempt cap; a retry budget: attempts per successful call, averaged over a window. The deadline is the part people skip. Attempt caps do not help when each attempt hangs for 30 seconds. Corrected model: retries are a load amplifier. Budget them like you budget requests. Myth 4: "If it failed, retry it" Not every error is retryable. Retrying a 422 just burns your window. Signal Retry? Why 400, 401, 403, 404, 422 No Your request is wrong. It will stay wrong. 408, 429 Yes Transient, or you are being throttled. 500, 502, 503, 504 Yes Server-side and usually temporary. Timeout, connection reset Yes, if idempotent You do not know whether it landed. 200 with a bad body No That is a validation bug, not a transport bug. Honor Retry-After when the server sends it. RFC 9110 defines the header. Guessing a backoff while the server hands you a number is just noise. Corrected model: classify first, retry second. A retry loop without a classifier is a random number generator with extra steps. Myth 5: "It succeeded, so the pipeline is healthy" A third-attempt success is a yellow flag. Nobody sees it, because you log outcomes, not attempts. Test: plot attempts per call over one day. If p95 attempts sits above 1, you are paying a retry tax. Alert on the ratio, not just on failures. A pipeline at 1.4 attempts per call is failing a third of the time and hiding it. Corrected model: success rate is a vanity metric here. Attempt rate is the health signal. The artifact: an attempt ledger Here is the wrapper I use. It writes one JSON line per attempt, so the retry history outlives the process. # attempt_ledger.py import hashlib, json, random, time import urllib.error, urllib.request RETRYABLE = {408, 429, 500, 502, 503, 504} def idem_key(payload: dict) -> str: blob = json.dumps(payload, sort_keys=True).encode() return hashlib.sha256(blob).hexdigest()[:32] def call_once(url, payload, key, timeout=10): req = urllib.request.Request( url, data=json.dumps(payload).encode(), headers={"Content-Type": "application/json", "Idempotency-Key": key}, method="POST", ) with urllib.request.urlopen(req, timeout=timeout) as r: return r.status, r.read().decode() def call_with_ledger(url, payload, ledger_path="attempts.jsonl", deadline_s=20, max_attempts=4, base=0.5, cap=8.0): key = idem_key(payload) t0 = time.monotonic() with open(ledger_path, "a") as ledger: for attempt in range(1, max_attempts + 1): start = time.monotonic() status, body, err = None, None, None try: status, body = call_once(url, payload, key) except urllib.error.HTTPError as e: status = e.code except Exception as e: err = type(e).__name__ ledger.write(json.dumps({ "key": key, "attempt": attempt, "status": status, "error": err, "elapsed_s": round(time.monotonic() - start, 3), "total_s": round(time.monotonic() - t0, 3), }) + "\n") ledger.flush() if status and 200