Why my incident agent says ‘we tried this before and it failed’.

It's 2 AM and checkout is returning 500s. The dashboard says the database connection pool is exhausted. The on-call engineer does what every runbook and every chatbot suggests: restart the connection pool. It doesn't work. It didn't work the last four times either. Somebody on the team knew that, but they were asleep, and the postmortem that said so was buried in a wiki nobody opens at 2 AM. That gap is what I built DejaVu to close. It's an incident-response agent that remembers every past incident: what happened, what the team tried, and which of those attempts actually worked. The feature I care about most isn't the one that tells you what to do. It's the one that tells you what not to do again. Repo: https://github.com/cybersleuth25/Deja-Vu Remembering successes isn't enough Most "AI with memory" projects store facts and retrieve the relevant ones. For incident response, that only gets you halfway. If the agent remembers that a database outage happened before, it'll happily suggest the fix that comes up most often in the text. In our history, the fix that comes up most often for pool exhaustion is "restart the connection pool", because people try it first every single time. And every single time, it fails. So memory has to carry verdicts, not just events. Every action taken during an incident gets recorded with a yes or no: did it work? The agent's job is then to turn those verdicts into a track record, and to say both halves out loud: "do this first" and "we tried this before and it failed". How memory works in DejaVu DejaVu uses Hindsight, the open-source agent memory system from Vectorize, as its long-term memory. (If you're new to the idea, Vectorize has a good explainer on what agent memory is.) There are three operations, and each has one clear job: Retain. When an incident is resolved, its postmortem goes into a Hindsight memory bank. When an engineer clicks Worked or Failed on an action, that verdict is retained too, tagged with the incident id and the action. Recall. When a new alert fires, the agent asks Hindsight for similar past incidents and pulls the incident ids out of the facts it gets back. Reflect. For the patterns panel, Hindsight's reflect summarises everything stored in the bank, the kind of overview a person would otherwise need an afternoon to write. Here is the real code that stores a verdict: def retain_verdict(self, inc: dict, action: str, worked: bool, note: str = "") -> None: self._call(self.client.retain, bank_id=self.bank_id, content=verdict_to_text(inc, action, worked, note), timestamp=datetime.now(timezone.utc).isoformat(), context="remediation verdict", tags=incident_tags(inc) + [f"action:{action}"], metadata={"incident_id": inc["id"], "action": action, "worked": "true" if worked else "false"}) One design decision mattered more than I expected. I put the incident id (like INC-0004) on every line of the text I retain. Hindsight extracts and consolidates facts from what you give it, so a fact that comes back from recall is not always word-for-word what you stored. Keeping the id inside the text means every recalled fact still maps back to one exact incident record. That's also why there are two stores. Hindsight does the remembering: finding incidents that are similar, even when the wording differs. A plain JSON file is the system of record that holds the exact counts. When the agent says "worked 3 of 4 times", I want that number to be exact, not a guess built up from fuzzy recall. Turning verdicts into a warning Once recall returns the similar incidents, ranking the actions is ordinary code, and I think that's a feature. Here is the core of it: smooth = lambda a: (a["worked"] + 1) / (a["tried"] + 2) do_first = sorted([a for a in actions if a["rate"] >= 0.5], key=lambda a: (-smooth(a), -a["worked"], -a["evidence"], a["action"]))[:3] avoid = sorted([a for a in actions if a["worked"] == 0 and a["tried"] >= 1], key=lambda a: (-a["tried"], a["action"])) Two small rules do most of the work: Smoothing. A fix that worked once out of one attempt shouldn't outrank one that worked 3 out of 4. Adding one success and one failure to every count, (worked + 1) / (tried + 2), keeps a single lucky result from jumping to the top. "Avoid" means never worked. An action only lands on the "we tried this before and it failed" list if it has been tried and has never once succeeded. That keeps the warning rare, and so people take it seriously. The LLM (I used openai/gpt-oss-120b on Groq) writes the short summary on top. It is only allowed to cite incident ids that recall actually returned. If it mentions an id that isn't in the matched set, DejaVu throws the text away and uses a plain template instead. I would rather show a boring sentence than an invented incident. Before and after Same alert, same data, memory switched off and then on. Memory off: Checkout 500s: DB connection pool exhausted on payments-api. Standard first steps: Restart connection pool; Restart service; Increase pool size. No incident history was consulted. That's the answer you'd get from any assistant. Note that its first suggestion is the one that has never worked for us. Memory on: This outage on payments-api matches 5 past incidents: INC-0004, INC-0009, INC-0014, INC-0020, INC-0029. Roll back pool config worked 3 of 4 times; Restart connection pool failed 4 of 4. The change "Tidy db config" touched db/pool.py 3h ago; a change to the same files preceded INC-0004, INC-0009, INC-0020. [Screenshot 1: the incident view with memory off, showing the generic steps] [Screenshot 2: the same alert with memory on, showing the Déjà vu badge, "Roll back pool config 3 of 4" under Do first and "Restart connection pool 0 of 4" under "We tried this before and it failed"] The second answer does three things the first can't. It names the fix that actually works. It warns you off the fix that has failed every time. And it points at a recent change to the same file that came before three of those outages. The same memory works for security incidents too. For a credential-stuffing attack from a known network, it recalls that per-IP rate limits failed 3 of 3 times and that blocking the attacker's ASN at the firewall worked 3 of 4. Watching it learn The clearest way to see the value is to give the agent more history and ask the same question each time: 0 incidents in memory: generic advice, and "restart the pool" comes first. 5 incidents: one match. "Roll back pool config" worked 1 of 1, "Restart connection pool" failed 1 of 1. 20 incidents: four matches. Rollback worked 3 of 4, restart failed 4 of 4. [Screenshot 3: terminal output of the learning-curve script at 0, 5 and 20 incidents] Nothing was retrained between those runs. The model is the same; only the memory grew. Every time an engineer clicks Worked or Failed, the next answer gets a little better. Limitations The traffic is simulated. DejaVu's detection rules run on generated events, not real production logs. The memory, ranking and recommendations are real, but connecting it to an actual log pipeline is still to be done. It's only as good as the verdicts. If nobody clicks Worked or Failed, there is no track record to learn from. The agent can't tell whether a fix worked on its own. It only recommends. DejaVu never blocks, rolls back or remediates anything itself. I consider that a feature, but it means a human still has to act. What I'd take away If you're building an agent with memory, store outcomes, not just events. "This happened" helps a little. "We tried X and it failed four times" is what saves the 2 AM engineer twenty minutes. Hindsight makes the remembering easy. The part worth designing carefully is what you choose to remember. Hindsight docs: https://hindsight.vectorize.io/ Hindsight on GitHub: https://github.com/vectorize-io/hindsight What is agent memory: https://vectorize.io/what-is-agent-memory Tags: @Code.in