My custom AI bot keeps giving bad advice because it cannot retain feedback from past calls

Your support bot tells a customer that a sync error goes away if they clear the app cache. It does not. The customer writes back, a person on the team finds the real fix, re-authorising the integration, and the ticket closes. Next Tuesday another customer reports the same error, and the bot recommends clearing the cache again, with the same confidence. The short answer: each call ends before anyone knows whether its advice worked, so the feedback arrives with nothing to attach to. Keep a record of every call with the ids of the stored notes the answer was built on. When the verdict comes in, from the user, the ticket or the person who took over, send it back as a rating on those notes, and have the bot recall from a memory that reads ratings when it ranks. Next time, the advice that failed and the advice that worked are no longer equals. Nothing is erased: the failed advice stays on record, with its outcome attached. Why the bot does not learn from its own calls A typical custom bot is a model call wrapped in retrieval: a system prompt, a search over help articles or stored notes, and the question. The verdict on last week's answer is in none of those. It is in the customer's reply and the helpdesk. Teams usually try three fixes, in this order. Edit the system prompt. Add "do not suggest clearing the cache for sync errors". A prompt is the right home for policy, but it scales like any document edited by hand: every bad call adds a line, nobody prunes them, and important rules compete with stale ones for the model's attention. Store answers and corrections in a memory. The correction now survives the call, but so does the cache advice, and similarity search ranks by closeness to the question. Both notes match, both come back, and the model picks one in context. A wrong note is as similar to the question after ten failures as after none. Attach the outcome to the note. When a call's result is known, record it on the notes that shaped the answer. The failed note now carries its failure and the working one its success, and a recall that weighs outcomes can tell them apart. Only this fix has a path from what happened back to what gets recalled next. What to keep from each call The answer is written during the call. The verdict arrives after it, sometimes days later, in another process. So the call must leave something for the verdict to find: Call record: what a bot keeps from each call so that a verdict arriving later can find what to judge: the question, the answer, and the ids of the stored notes the answer was built on, under a key the verdict will carry. The ids are per note, not per conversation. One piece of advice failed, not the exchange. If the model can say which recalled notes it relied on, keep those; otherwise keep every note that was in the prompt and accept that they move together. The key is one the verdict will carry: a ticket id, a thread id, the session id your feedback button already sends. When one key covers several answers, the code below keeps the latest. Keep call records in your own database, apart from the notes the bot recalls. Written into the same memory, every logged answer becomes one more candidate in the next recall, the bad ones included. Where the verdict comes from Explicit ratings are rare. Other outcomes are often already recorded somewhere, and wiring those places in works better than waiting for a thumbs-down. signal when it arrives what it says about the advice what to send thumbs up or down on the answer seconds later, if at all how the answer looked, often before it was tried positive or negative the user says it did not work the next message, or the next day it was tried and it failed negative a person takes over and solves it another way when the handover closes it failed, and here is what worked negative, plus the fix as a new note the same user returns with the same problem hours or days later the first answer did not hold negative, smaller the ticket closes and stays closed after a quiet period you choose it probably worked positive, smaller nothing happens never nothing nothing Two rows need care. A handover for a reason the bot was never meant to handle, a refund or an angry customer, says nothing about the advice. And silence, the commonest row, is not a verdict. The third row does the most work. A rating can lower the advice that failed, but only a note can supply the advice that worked, so the person's fix goes into the memory the bot recalls from. What a rating changes, and what it does not A rating cannot change the answer already given. It changes the order of what the next recall brings back. Keep the failed note rather than deleting it. Clearing the cache is right for a different error, and a deleted note cannot answer "why did the bot tell customers to clear the cache in October?". Ratings tune ranking, not truth. If a help article is wrong, fix the article. The loop needs no particular product: a list of failed advice the bot checks approximates it, a store you run can carry an outcome field, a managed memory layer keeps the bookkeeping. What matters is the path from outcome back to recall. A minimal version in code This uses mnemoverse 0.4.0, the Python SDK of Mnemoverse (the author's company), set up as on the Python SDK page. The model call is a placeholder, and call records live in SQLite. import json import sqlite3 from mnemoverse import MnemoClient memory = MnemoClient() # reads MNEMOVERSE_API_KEY DOMAIN = "bot:support" db = sqlite3.connect("calls.db") db.execute("CREATE TABLE IF NOT EXISTS calls " "(call_id TEXT PRIMARY KEY, question TEXT, answer TEXT, used TEXT, concepts TEXT)") def answer(call_id: str, question: str) -> str: recalled = memory.read(question, domain=DOMAIN, top_k=5) reply, relied_on = bot.reply(question, recalled.items) # your model call used = [str(item.atom_id) for item in relied_on] db.execute("INSERT OR REPLACE INTO calls VALUES (?, ?, ?, ?, ?)", # latest answer per key (call_id, question, reply, json.dumps(used), json.dumps(recalled.query_concepts))) db.commit() return reply def on_verdict(call_id: str, outcome: float, fix: str | None = None) -> None: row = db.execute("SELECT question, used, concepts FROM calls WHERE call_id = ?", (call_id,)).fetchone() if row is None: return # no call record: the verdict has nowhere to land question, used, concepts = row[0], json.loads(row[1]), json.loads(row[2]) if used: result = memory.feedback(atom_ids=used, outcome=outcome, # -1.0 to +1.0 query_concepts=concepts, domain=DOMAIN) print("updated:", result.updated_count) if fix: # what actually worked, from the person who solved it note = memory.write(f"Problem: {question}\nWhat worked: {fix}", domain=DOMAIN) if not note.stored: print("Not stored:", note.reason) on_verdict("ticket-1042", -1.0, fix="Re-authorise the integration under Settings, Connections.") on_verdict("ticket-1043", 1.0) # the customer confirmed it worked feedback gets the same domain as the read; the SDK page notes that without it the SDK sends "general". A write that is too similar to the nearest memory in the same domain is not stored, so the code checks stored. The API docs state the effect in one line: "Ratings re-rank what comes back next." The overview says what stays: "feedback changes the order of the next read, not the stored text". A bot on an MCP client rates through the memory_feedback tool instead. Check it with a control Seed notes about two different problems, answer one question about each through answer(), and check that the two call records share no id. Print the valence of the notes each call relied on, "Outcome polarity [-1, +1]" in the API reference, and stop the process. In a new process, send a negative verdict for the first call only. updated_count, "Number of atoms updated" in the reference, should equal the number of ids recorded for that call. If it is zero, check the ids and the domain first. Read both questions again and print valence for the same notes. What to look for is the first call's notes moving toward -1 since step 1. Control: the second call got no verdict, so its notes are the comparison for step 3. If they moved and the first call's did not, look at the key the verdict was filed under. If both moved alike, step 3 does not show the verdict's effect. The library page follows the same loop for coding agents, where the verdict is a build or a test run, with the limits of outcome memory in more detail. Where does the verdict on your bot's answers end up today?