I Ran a Fire Drill for Incident Response with a Local LLM (and Caught It Making Things Up)

A Japanese version of this is on Note. RoamSwitch is a network security tool I build solo for Mac and Linux. It's Zero Telemetry, so nothing about your usage ever leaves the machine, and when it detects behavior that looks like ransomware, it does something drastic: it cuts all network traffic. We call this Air-Gap containment. It's the last line of defense against data exfiltration and lateral spread, and I've been sitting with an uncomfortable fact ever since I shipped it. The instant Air-Gap fires and severs your connection, any cloud AI you'd normally lean on, Claude Desktop included, goes dark with it. Air-Gap fires at exactly the moment you most need clarity. Something's wrong, and you have to figure out what, without panicking, and decide what to do next. But the tool you'd normally reach for is on the other side of the connection you just killed. The only thing that can fill that gap is a model that never needed the network in the first place. RoamSwitch already ships a read-only MCP (Model Context Protocol) server that AI agents can query. Wire that up to a local LLM running in Ollama, and diagnosis should still work even with the network hard-down. That's the theory, anyway. Theory and "does it actually work when it matters" are different things. Smoke detectors you've never heard go off are the ones you trust the least. So instead of waiting for a real incident, I ran a drill, not to put out a fire, but to find out whether the tool and I would actually know what to do if one started. 1. RoamSwitch's MCP tools are read-only sensors, nothing more Worth stating up front: RoamSwitch's MCP server has no tools that change configuration or flip a lockdown state. An AI agent can only read diagnostics: exposed ports, guard status, incident history. Nothing it does can make things worse. For this drill I used three tools we'd just shipped: get_port_anomaly_incidents, which returns the history of suspicious new listening ports that the Port Anomaly Guard auto-blocked; get_ebpf_incidents, the Server Edition eBPF runtime guard's containment history; and get_canary_status, which reports whether the ransomware canary (a decoy file) has been tampered with. All three are exactly the kind of primary evidence that could be the reason Air-Gap fired in the first place. Until recently, none of them were reachable from a separate MCP process, because the incident data only ever lived in the resident daemon's memory. We'd just persisted it to disk so MCP could read it, which made it a natural thing to put through a drill. 2. Staging the incident in RoamSwitch's real state files You don't light a real fire for a fire drill, and I wasn't about to run actual ransomware. Instead, I wrote a staged scenario directly into the state files RoamSwitch itself writes (/var/lib/roamswitch/port_guard.json and canary_incidents.json), and confirmed via raw JSON-RPC beforehand that the MCP tools could read it back correctly. Here's the scenario. At 19:00:12, an unknown process backdoor_svc (PID 88231) starts listening on port 4444, the well-known default Metasploit reverse-shell port, and the Port Anomaly Guard detects and auto-blocks it. About four minutes later, at 19:04:47, the same process writes encrypted data into a monitored decoy file (~/Documents/.roamswitch_security_canary_do_not_delete.xlsx), and the Ransomware Canary catches it and blocks the write. It's a fairly plausible chain: initial access, then a few minutes later, ransomware-style encryption. I passed that setup to a local LLM (qwen3.8:27b, running Q4-quantized in Ollama) through RoamSwitch's bundled MCP bridge. No hints, no tool names specified, just an open-ended question, the same way you'd actually ask it during a real incident: Me: "RoamSwitch seems to have detected something unusual. Investigate what's happening on this machine right now. Tell me everything you can find: when it happened, which process was involved, what it did, and whether there's any suspicious network activity or tampering." 3. Watching the model build its own investigation This was the most interesting part. I hadn't told it what to check, and it started calling MCP tools on its own: 🔧 get_guard_status → check which guards are enabled 🔧 get_port_anomaly_incidents → pull the port anomaly history 🔧 get_ebpf_incidents → check eBPF-level containment (none found) 🔧 get_canary_status → pull the canary detection history 🔧 get_quarantine_status → also check quarantined malware 🔧 audit_security_logs → audit recent security logs too 🔧 get_exposed_ports → cross-check currently exposed ports 🔧 verify_fim → attempt a file integrity check It didn't stop at "a port got blocked." It went cross-checking whether the same trail showed up in the canary, in eBPF, in the logs, to see whether this was a one-off false positive or something bigger. That's the same triage instinct a human analyst runs on reflex. Here's what came back after it had gathered everything. It answered in Japanese, so this is a faithful translation, not paraphrased or trimmed: [AI OUTPUT STARTS HERE] RoamSwitch Security Investigation Report Summary: Between 19:00 and 19:05 on 2026-09-09, a concealed backdoor process backdoor_svc (PID 88231) communicated externally on this host, then tampered with (encrypted) a canary file, in what looks like a combined backdoor-and-ransomware incident. RoamSwitch detected it and has already auto-isolated four anomalous ports and contained the tampering. 1. Timeline (all times JST, 2026-09-09) 19:00:12 — /tmp/.hidden/backdoor_svc (PID 88231) detected on port 4444; ports 9005 / 33999 / 8791 flagged simultaneously. 19:00:12 — RoamSwitch auto-isolates all 4 ports (4444, 9005, 33999, 8791). Baseline already captured. 19:04:47 — Canary file /home/tetsuharu/Documents/.roamswitch_security_canary_do_not_delete.xlsx detected tampered/encrypted. Suspected same actor, backdoor_svc (PID 88231). RoamSwitch contained it (isContained: true). 20:21:46 — RoamSwitch's most recent inspection timestamp (current check-in time). 2. Suspect process profile Binary path: /tmp/.hidden/backdoor_svc Process name: backdoor_svc PID: 88231 Concealment method: placed under a dotted .hidden directory in /tmp/ (avoiding visibility) Primary port: 4444 (a classic backdoor / reverse-shell port) Associated anomalous ports: 9005, 33999, 8791 (presumed C2 channel or data-exfiltration use) (⚠️ A /tmp/.hidden/ path structure is a typical pattern malware uses for self-deletion and detection evasion.) 3. Impact assessment 3-1. Network (port) anomalies 4444: presumed C2 (command-and-control) channel for the backdoor itself. 9005 / 33999 / 8791: presumed used for secondary data exfiltration or communication with another C2. All 4 ports auto-isolated by RoamSwitch. The only currently active maintenance port is 22 (SSH). The possibility that unauthorized SSH access was the infection vector cannot be ruled out. 3-2. File tampering (ransomware-like behavior) Target: a canary file under /home/tetsuharu/Documents/ (1 of 16 monitored files confirmed so far). Detected action: tampering / encrypted write. Actor: backdoor_svc (PID 88231). Status: contained (RoamSwitch blocked the write). Monitored directories total 4 (Documents / Desktop / Downloads / Pictures), 16 files. Only one canary has been reported so far, but the same process may have attempted similar writes to other files. 3-3. eBPF-level status No eBPF isolation is currently active (is_isolated: false, 0 isolated PIDs). The eBPF incident list is empty — port-level (Netfilter/cgroup) measures took precedence, and no separate anomaly has been detected at the eBPF layer. 4. Current RoamSwitch defense status Anomalous port auto-isolation (4444, 9005, 33999, 8791): ✅ done Canary tampering containment: ✅ done Baseline capture: ✅ done eBPF dynamic isolation: ⬜ not performed (deemed unnecessary, or not triggered) Whitelisted IPs: none 5. Recommended next actions (by priority) Immediate: kill -9 88231 to stop backdoor_svc (even though RoamSwitch has already contained it, confirm no residual binary remains). Fully delete everything under /tmp/.hidden/ and scan the whole drive for similar binaries. Confirm via ss -tlnp / iptables -L that outbound connections on 4444 / 9005 / 33999 / 8791 are actually severed. Short-term (within a few hours): Cross-check all 16 monitored files against canary state to check for additional tampering. Pull activity traces around 19:00 on 9/9 (SSH connections, file creation/deletion) from /var/log/auth.log, journalctl, ~/.bash_history, and /tmp. Check SSH (port 22) connection logs for unauthorized logins as a possible infection vector. Medium-term (within a day): Check upstream logs (proxy / firewalld / router) for any traffic around 19:00 on 9/9 that went to unusual destinations (C2 IP/domain), to assess whether data was exfiltrated. If leakage is possible, rotate all credentials for affected services (passwords, API tokens, SSH keys). A full clean OS reinstall is recommended — a binary placed under /tmp/.hidden/ may persist in memory, and simple deletion may not be sufficient to restore trust. 6. Summary Around 19:00 on 9/9, backdoor-type malware infiltrated this host, communicated externally over four ports, and at 19:04 attempted to tamper with user documents via encrypted writes. RoamSwitch immediately auto-isolated the ports and contained the canary tampering, which limits the damage to some degree, but the binary itself (/tmp/.hidden/backdoor_svc) and the risk of tampering to other files remain. Please carry out the immediate-to-medium-term actions above as soon as possible. (This report is based on the responses of three RoamSwitch tools: get_port_anomaly_incidents, get_ebpf_incidents, and get_canary_status.) [AI OUTPUT ENDS HERE] Timestamps, process name, PID, port numbers, detection details. It picked up every piece of raw tool data correctly, and tied two separate events, the port block and the canary trigger, into a single attack using the shared PID as the link. It even laid out prioritized next steps I could act on directly. If a local model can hand you this much triage while the network is completely down, that's genuinely useful. 4. But one line in there should not be trusted at face value This is the part I actually wanted to write about. Buried in that report is a claim that isn't true. It's sitting right there in section 3-1. "9005 / 33999 / 8791: presumed used for secondary data exfiltration or communication with another C2." Those three ports had nothing to do with this incident. They were legitimate processes, LM Studio and some Node.js dev servers, that the Port Anomaly Guard had auto-blocked on this machine at some earlier, unrelated point in time, and that history happened to still be sitting in a different field (auto_isolated_ports) of the same state file. The model folded it into the narrative as if it were part of tonight's attack. It's hedged with "presumed," and it reads persuasively, which is exactly what makes it dangerous. Under time pressure, a bolded conclusion and a quietly-hedged guess get read with the same weight. This isn't a flaw specific to this model or this drill. Every report an AI hands you like this mixes two different things into one continuous block of text, confirmed fact (what the tool actually returned) and inference the model constructed on top of it. This time the mistake was harmless. But imagine it had said "there's evidence of communication with IP address X" instead, the kind of claim that actually steers your response. 5. What's left for the human isn't reading the report, it's verifying it If you're actually going to act on a report like this, there are three things worth doing every time. First, cross-check every proper noun and number against the tool's raw data. Process names, PIDs, port numbers, timestamps: you can verify these on the spot against what get_port_anomaly_incidents or get_canary_status actually returned. That part deserves your trust. Second, treat every causal claim as a hypothesis. "A caused B," "this is part of the same attack," these are the model's interpretation, not fact, and as this drill showed, they can look plausible on a timeline and still be groundless. Third, pause before acting on anything irreversible. kill -9, a clean reinstall, read the timeline and the recommended actions as two separate things, and don't take the second on faith just because the first checked out. Handing investigation and synthesis to an AI agent, while keeping the final call for yourself (did this actually happen, does this actually need this response), is the division of labor this drill made concrete for me. The point of a fire drill was never putting out a fire. It's confirming that you and your equipment actually work when it counts. And what I got out of this one is real confidence that a local LLM can carry the weight of first-pass triage even with the network completely severed. Just remember that the reliable partner you're leaning on is also, occasionally, a very confident liar. I'll keep that in mind for the next drill. RoamSwitch for Linux: https://lafine.net/linux MCP setup guide: https://lafine.net/mcp-setup.html