Microsoft could not fix its own outage on the first try. Is your team ready when your vendor goes down next?
Microsoft could not fix its own outage on the first try. Is your team ready when your vendor goes down next? On 12 November 2026, Microsoft 365 went down. Exchange Online (EX1550543) opened at 16:17 UTC, and the wider Microsoft 365 incident (MO1550821) followed about two hours later. Outlook, Teams, SharePoint, Exchange Online and Copilot were all affected. The outage stayed degraded for more than a day. The facts below are Microsoft's own published words (Microsoft Service Health, status.cloud.microsoft, verified 14 November 2026): Many servers became "unexpectedly unavailable" after failing automated health checks. Workload shifted to the remaining servers and overloaded them. Not a traffic surge. No sign of cyberattack. The first fix did not take effect as expected. Microsoft's own wording was "the change is not taking effect as expected". Update 17:57 UTC 13 November 2026: "This was a technical issue not related to cyber or AI / agents. The incident was introduced by an issue during an earlier network capacity upgrade that surfaced following a recent network change. We have addressed the underlying issue and continue to restore service to remaining customers and regions." Microsoft extended the email timeout window to 48 hours during the incident. Final update 04:17 UTC 14 November 2026: MO1550821 reads "serviceRestored", recovery confirmed across previously affected environments, with end time given as 16:10 UTC 13 November 2026. The full post-incident report has not yet been published. While Microsoft worked, every customer at once had no self-fix: no local setting to change, no server to restart, nothing to do but wait. The operator question is not "why did Microsoft fail". It is: when MY critical vendor goes down next, is my team ready? Do we know the vendor status page is step 1 of the runbook? Is there a fallback channel that was actually tested? Is there an escalation that does not depend on the down vendor? Is there a comms template that states only the vendor's published cause? Does anyone know the mail timeout window? I built a small tool that answers it from your own files. It is called outagekit, and it is a local, zero-dependency TypeScript CLI. MIT licensed. No server, no network except one documented liveness check, no upload, no API calls. It reads your runbooks and a small manifest, and reports whether your vendor-outage plan is complete - with the files and lines that prove it, a deterministic verdict, and a CI exit code. The problem: status monitors tell you the vendor is down. Nobody checks if your plan is ready Status monitors tell you the vendor IS down. Incident platforms run the incident once it starts. Neither answers, from your own files, whether your readiness plan is complete BEFORE the incident. So a team can have a runbook that tells someone to reinstall Outlook during a vendor outage, a fallback channel that was last tested seven months ago, an escalation where the primary and backup both use the same phone channel, and a staff announcement that invents its own root cause - and nobody finds out until the vendor is already down. outagekit audits the plan itself, before that day comes. What it does outagekit runs ten checks, C1 to C10, over your manifest and runbooks: # Check Rule C1 STATUS-PAGE Every critical vendor has a statusPageUrl configured C2 STATUS-PAGE-LIVE A HEAD request to the status page URL returns 2xx/3xx (GET retry when HEAD is refused; offline pages report UNCHECKED, never PASS) C3 RUNBOOK-EXISTS A runbook file exists for every critical vendor C4 RUNBOOK-STEP-1 The runbook names the vendor's status page URL within its first 40 lines C5 NO-LOCAL-FIX The runbook contains no forbidden local-fix step (reinstall, rebuild, reset password or credentials, change DNS records, re-create the mailbox or profile) C6 FALLBACK-CHANNEL A fallback comms channel is defined on a channel that does not depend on any listed vendor, and lastTested is within 90 days C7 ESCALATION Primary and backup escalation contacts defined, on different channels, and not the same person C8 COMMS-TEMPLATE Staff-announcement template exists, contains a {{vendor_cause}} placeholder (state only the vendor's published cause), and asserts no root cause of its own C9 MAIL-TIMEOUT Mail timeout / retry window is documented and at least 24h C10 DRILL Last full vendor-outage drill within 180 days The verdicts are plain: READY (no HIGH or CRITICAL findings), GAPS (HIGH findings, no CRITICAL), NOT-READY (any CRITICAL finding). WARN findings never change the verdict alone. The answer comes back as an exit code a shell or CI step can use directly: 0 = READY, 1 = GAPS, 2 = NOT-READY (or a config error). The thresholds (90-day fallback test, 180-day drill, 24h mail timeout) are defaults chosen by the author, published in outagekit.json, and overridable per repo. They are not industry standards, and no standard is cited for them. outagekit init # scaffold outagekit.json + runbook + template outagekit audit # report (exit 0 always) outagekit audit --format json # full evidence: findings with file + line where the finding comes from a file outagekit check # CI gate: exit 0/1/2 outagekit explain --check C5 outagekit diff --baseline audit.json The demo: one team that is not ready, one that is The GIF above is a real terminal session, not a mockup. It audits examples/acme-ops, a synthetic team whose plan has real gaps, and examples/ready-ops, a synthetic team whose plan is complete. Both fixtures are invented companies with invented dates - the verdicts are the fixtures' own, not measurements of any real company. $ outagekit audit examples/acme-ops outagekit audit - examples/acme-ops verdict: GAPS (CRITICAL 0, HIGH 8, WARN 3) check name result severity evidence ---------------------------------------------------------------------------------------------------- C1 STATUS-PAGE WARN WARN [Stripe] important vendor "Stripe" has no statusPageUrl configured C2 STATUS-PAGE-LIVE PASS - C3 RUNBOOK-EXISTS PASS - C4 RUNBOOK-STEP-1 FAIL HIGH [Microsoft 365] runbook does not name the status page URL in its first 40 l... (runbooks/microsoft-365.md:1) C5 NO-LOCAL-FIX FAIL HIGH [Microsoft 365] forbidden local-fix step: "reinstall" (runbooks/microsoft-365.md:12) HIGH [Microsoft 365] forbidden local-fix step: "reset the password" (runbooks/microsoft-365.md:14) HIGH [Microsoft 365] forbidden local-fix step: "re-create the mailbox" (runbooks/microsoft-365.md:14) C6 FALLBACK-CHANNEL WARN WARN fallback channel last tested 201 days ago (limit 90 days) C7 ESCALATION FAIL HIGH primary and backup escalation use the same channel ("phone") - one dead cha... C8 COMMS-TEMPLATE FAIL HIGH comms template has no {{vendor_cause}} placeholder - it cannot state only t... (templates/staff-announcement.md) HIGH template asserts a cause of its own ("was caused by") instead of the vendor... (templates/staff-announcement.md:5) HIGH template asserts a cause of its own ("root cause is") instead of the vendor... (templates/staff-announcement.md:6) C9 MAIL-TIMEOUT PASS - C10 DRILL WARN WARN last full vendor-outage drill was 197 days ago (limit 180 days) summary: 0 CRITICAL, 8 HIGH, 3 WARN, 0 UNCHECKED, 0 SKIPPED Three of the eight HIGH findings are the same mistake in different words: the runbook tells someone to fix a vendor outage locally. "Reinstall Outlook to fix the mailbox." "Reset the password and re-create the mailbox." During a vendor outage there is no local fix - no reinstall, no rebuild, no password reset, no DNS change, no mailbox re-creation. The runbook is the wrong plan, and the audit names the exact lines. outagekit explain shows why a check scored the way it did, with the rule, every finding with file and line, and how to fix it: $ outagekit explain examples/acme-ops --check C5 explain C5 - NO-LOCAL-FIX result: FAIL (verdict GAPS) rule: The runbook contains no forbidden local-fix step (reinstall / rebuild / reset password or credentials / change DNS records / re-create the mailbox or profile). A negation (do not / never) before a phrase in the same clause is not a hit; a phrase after a clause break (comma, semicolon, but) is still flagged. The per-runbook allowlist covers documented exceptions. findings (3): [HIGH] forbidden local-fix step: "reinstall" (runbooks/microsoft-365.md:12) evidence: Step 3: Reinstall Outlook to fix the mailbox. [HIGH] forbidden local-fix step: "reset the password" (runbooks/microsoft-365.md:14) evidence: Step 4: Reset the password and re-create the mailbox. [HIGH] forbidden local-fix step: "re-create the mailbox" (runbooks/microsoft-365.md:14) evidence: Step 4: Reset the password and re-create the mailbox. how to fix: Remove the local-fix step. A vendor outage has no local fix: no reinstall, no rebuild, no password reset, no DNS change, no mailbox re-creation. If the step is a documented exception, add the exact phrase to the runbook allowlist in outagekit.json. The same commands on the ready fixture tell a different story: $ outagekit audit examples/ready-ops outagekit audit - examples/ready-ops verdict: READY (CRITICAL 0, HIGH 0, WARN 0) $ outagekit check examples/ready-ops check: READY (exit 0) outagekit diff compares an audit against a saved audit --format json baseline and reports NEW / FIXED / STILL-OPEN findings, so readiness regressions show up in CI before they become an outage. A --max-warn N flag fails the gate when the WARN count exceeds N. The test suite is 50/50 passing (39 unit + 11 e2e), and the audit over the two demo fixtures measured a median of 0.62 ms and p95 of 1.9 ms (200 runs each, liveness skipped). Install (from source only) The npm package is not published yet, so for now install from source: git clone https://github.com/techdevfun/outagekit.git cd outagekit npm install npm run build Requires Node.js 20 or newer, zero runtime dependencies. A clean install from source was measured on 14 November 2026 at 3.7 seconds (npm install + npm run build, no cache). The golden path (init then audit then check) measured 0.4 seconds with a fast connection - each audit does live C2 HEAD requests, so the time is network-dependent. Run it from the repo root, or npm link to make outagekit available everywhere. Privacy: what outagekit never touches It reads runbook text and the manifest only. It never reads secret values or env values, and never reads files matching .env*, *secret*, *.pem or *.key. Escalation contacts are names and channel TYPES only. A manifest that puts a phone number or email address in an escalation contact is refused at load time, and evidence lines in output are redacted for email addresses and phone-like digit runs. JSON output carries per-check findings with file and line where the finding comes from a file - never file contents beyond the matched line. The only network call is the documented C2 liveness check of user-configured status-page URLs. No upload, no API calls, no telemetry. Where the rivals beat us Honesty first: outagekit audits a plan on paper. It does not watch the vendor, and it does not run the incident. Statuspage / Better Stack give REAL-TIME vendor status and alerts. outagekit only checks the status-page URL is configured (and, when online, reachable). If you need to know the moment a vendor goes down, they are the better tool for that job. incident.io / PagerDuty page real humans and run real incidents. outagekit never pages anyone. They start when the incident starts; outagekit checks the plan before it starts. HA/DR Field Kit covers failover-triage depth (databases, Applications, Incident Command, Dashboards) outagekit never touches. It is manual browser checklists - no vendor-outage scope, no manifest, no CI gate, no freshness tracking. OutagePulse debugs YOUR outage after it happens, over MCP. It never audits your readiness plan for a VENDOR outage, and it needs a cluster where outagekit needs only files. Status-page aggregators / Downdetector give crowd reports that a vendor is down. A signal is not a plan; no audit, no gate, no evidence. The strongest rival pair is incident.io plus Statuspage: together they cover "know the vendor is down" and "run the incident". Neither covers "prove, from your own files, that the plan is ready before the incident". outagekit only sees what is in the files it reads. The team's actual skill, the vendor's real recovery speed and the human judgement in the room are invisible to it. Related, from the same author: vendormap measures vendor CONCENTRATION from the repo - which single vendor can take you down. Concentration and readiness are complementary halves, not the same tool. Honest limits Checks are readiness HEURISTICS over your own files, not risk measurements. A READY verdict means the plan is complete on paper, not that the next outage will go well. C5 (NO-LOCAL-FIX) is a phrase heuristic. It can miss a local-fix step worded differently, and it trusts the allowlist for documented exceptions. A negation ("do not", "never") before a phrase in the same clause is not a hit; a phrase after a clause break is still flagged. C2 checks that a status page answers, not that the vendor is healthy. A page can answer 200 while the vendor is down - that is what the page is for; outagekit only verifies the URL is the right one to check. Thresholds (90-day fallback test, 180-day drill, 24h mail timeout) are author defaults, not industry standards. The tool never claims a vendor is "safe" or "risky", never states a root cause beyond the vendor's published wording, and never claims an incident is "fully resolved" beyond the vendor's own "serviceRestored" wording. Why this shape A vendor's bad week is not a hypothetical. This week the largest enterprise vendor in the world could not fix its own outage on the first attempt, and its customers could do nothing but wait. The question worth answering is not what happens to the vendor next. It is whether your team has a plan that survives the wait - a status page as step 1, a tested fallback channel, an escalation that does not depend on the down vendor, and a comms template that states only what the vendor published. If your team depends on a critical vendor, run one audit before the next bad day: github.com/techdevfun/outagekit Feedback welcome in the repo issues. The roadmap (generic webhook notification on NOT-READY, pre-commit hook example, enterprise rollout docs) is in the README.
This is a summary aggregated from Dev.to. Read the complete article on the original site:
Read full article at Dev.to