I Rotated 4 Secrets in 6 Days. 3 of Them Broke Production.

Every guide on secret rotation ends with "and then update the consumers." That sentence is where production dies. I run two self-hosted n8n instances for an automation studio in Israel. This morning I counted what they hold: 114 credentials, referenced from 1,311 nodes across 260 workflows. The single most-used credential (a Postgres connection) sits in 233 nodes in 80 workflows, 40 of them active. The WhatsApp gateway key is in 106 nodes. Between August 20 and August 25 I rotated four secrets. Three of them took something down. Each one failed in a different way, and none of the three failures was inside n8n. That is the part worth writing down. Failure 1: the copy you forgot (2.5 days, zero alerts) August 20. I rotated a Chatwoot API token. The token is used by a WAHA→Chatwoot bridge, and the bridge keeps its own copy of the token per inbox app. Six apps, six copies. I updated four. The two inboxes on the old token went quiet. Inbound messages hit POST /contacts/filter, got 401 Invalid Access Token, and vanished. Outbound looked fine from the agent's side (the row was created in Chatwoot), then stalled on the same 401. No alert fired anywhere, because nothing was "down": the process was up, the endpoint answered, the queue just filled with failed jobs. I noticed on August 23. Two and a half days. The side effect was worse than the outage. Looking up a contact needs the token; creating one goes through a public endpoint that doesn't. So every inbound message during the outage created a fresh contact_inboxes row. When I finally counted, the table held 18,489 duplicates (some older, most from this), and those later produced 404s on update_last_seen, because the bridge picked one mapping and the conversation sat on another. Fix that stuck: a daily check that validates every bridge app's token against Chatwoot, one line per app, non-zero exit if any fails. The hash comparison runs without printing a value: sha256(config.accountToken) against left(encode(sha256(token::bytea),'hex'),16) in access_tokens. Failure 2: the derived value (14 hours of rejected calls) Same night, August 20, 22:55. I rotated the signing key of the Hebrew AI call-answering service I run. The key signs magic links and admin cookies, which I knew. It also derives the URL token that Telnyx uses to deliver inbound calls to /voice/telnyx/, which I had forgotten. Telnyx kept posting to the old URL. Every inbound call got 403 bad URL token until 13:07 the next day. Six calls from four numbers, rejected by a service whose only job is answering calls. Nothing in the rotation touched Telnyx, so nothing in it could have warned me. Three layers now: on every boot the service compares its webhook URL with what Telnyx has and corrects it; any 403 on the token triggers an immediate resync plus a Telegram alert; and a fallback URL at Telnyx points to an n8n workflow on a different server that plays an apology recording. Detection gap went from 14 hours to seconds. I verified that by firing a call at it. Failure 3: the shadow override (858 × 401 in 38 hours) August 23. WAHA API key. This one lives in six places: two lines in the .env, an nginx snippet that injects the key on the public media path, two n8n credentials, and the local keychain. I had a tested procedure for all six. The procedure updated the nginx snippet. But the vhost file had a second, hardcoded proxy_set_header X-Api-Key ; line inside the /api/files/ location, left over from before the snippet existed. The hardcoded line won. The snippet variable was correct, and nothing read it. Result: 858 requests returned 401 over 38 hours, and every media message (voice notes, images, stickers) on five WhatsApp sessions stopped syncing to Chatwoot. In the inbox this shows up as "unsupported message type" notes, not as errors. The public vhost stayed green the whole time. Nothing on it fetches media except real clients. The verification I use now: from inside the WAHA container, curl a real media path through the public domain. Localhost 200 plus public 401 means an injection layer is holding an old value. The one that worked (4 destinations, verified in order) August 25. A client website's webhook secret, after it leaked: the webhook trigger node stores the full request headers in execution data, X-Webhook-Secret included, and I had opened that execution to debug something else. Four destinations: a config row in Supabase, a file on the client's WordPress host, an n8n credential, and the keychain. What was different: I wrote the verification before the rotation. New secret → the endpoint returns 200. Old secret → 401. Then wait for the two scheduled workflows that consume it (a 2-minute and a 5-minute cron) to log success. Only then delete the old value. It took longer than the other three. It was also the only one where I didn't have to explain anything to a client. The pattern Three failures, three different mechanisms: Failure Where the stale copy lived What was silent Chatwoot token 2 of 6 bridge apps Inbound messages dropped, no alert Signing key A value derived from it, registered at a third party Inbound calls rejected at the carrier WAHA key A hardcoded override under the variable I updated Media sync, while health stayed green The rotation itself was never the problem. The inventory was. None of my consumer lists was complete, and two were wrong in ways that only showed up after the old value stopped working. What I run before touching a credential now The n8n side of the inventory is a query. Workflows store nodes as JSON, and each node with a credential carries credentials: { : { id, name } }. This flattens that into a blast-radius table: WITH n AS ( SELECT w.id AS wid, w.active, w."isArchived" AS arch, jsonb_array_elements(w.nodes::jsonb) AS node FROM workflow_entity w ), c AS ( SELECT wid, active, arch, e.key AS ctype, e.value->>'id' AS cid FROM n, LATERAL jsonb_each(node->'credentials') e WHERE node ? 'credentials' ) SELECT c.cid, coalesce(ce.name, '') AS name, c.ctype, count(*) AS nodes, count(DISTINCT wid) AS workflows, count(DISTINCT wid) FILTER (WHERE active AND NOT arch) AS active_workflows FROM c LEFT JOIN credentials_entity ce ON ce.id = c.cid GROUP BY 1, 2, 3 ORDER BY nodes DESC; Names and counts only. It never touches the data column. Three things it told me this morning that I didn't know: 8 credential IDs are referenced by workflows but no longer exist in credentials_entity. The rows. All in inactive or archived workflows, but one of them is a Telegram credential still referenced from 43 nodes. 44 of the 100 credentials on the main instance were modified in the last 30 days. "We rotate rarely, so let's do it carefully" was the wrong mental model. We rotate constantly. It has to be routine. The Postgres credential with 233 references is a single point of failure that no dashboard shows. If I ever rotate that password, the order of operations matters more than the password. For the update itself: on n8n 2.36 the public API accepts PATCH /api/v1/credentials/{id} with a data object. I checked the route this morning with a body it had to ignore and got a 200 back. Ten days ago I was still creating a new credential, rewriting every node reference, and deleting the old one: three writes and a chance to miss a node. One PATCH, then confirm versionId equals activeVersionId on each consuming workflow, because a saved workflow is not necessarily the running one. The secret you already leaked One more thing from that fortnight, because it changed how I build the AI agents I build for small businesses. An insurance agency's reply bot pulled its API key from a config table with a Postgres node: SELECT value FROM config WHERE key = 'api_key'. Clean, no hardcoding. It sat in every execution instead. The node's output JSON contained the value, and n8n saves node output to execution_data. When I went looking: 980 rows held that key, and 649 held a second one (a Chatwoot token loaded the same way). Every time anyone opened an execution to debug it, the secret was on screen. That is how it leaked to me. The fix is to keep the secret out of node output entirely. httpCustomAuth merges its body straight into the request (requestOptions.body = { ...requestOptions.body, ...customAuth.body } in HttpRequestV3), and httpHeaderAuth does the same for a header. Drop the column from the SELECT, attach the credential to the HTTP node, and the value never enters JSON. Then scrub what is already stored: UPDATE execution_data SET data = replace(data, :'sec', '***REDACTED***') WHERE position(:'sec' IN data) > 0; (Run it from a script that reads the value from stdin, so it doesn't land in your shell history either.) The question For your single most-referenced credential: how many copies of it exist outside the system that owns it? Bridges, proxies, a file on a client's host, a URL registered at a third party, a derived token. If you have the number, I'd like to hear how you keep it current. If you don't have the number, that is the number.