I Thought This Was a Classification Problem. It Wasn't.

Previously: 9 Bugs That All Looked Like a Working System · I Built an AI That Rewrites Its Own Prompts · The Edit That Fixed 4 Tasks and Broke 1 · I Let an LLM Rewrite Its Own Prompt. The Real Win Was the Gate That Rejected It. · I Tried 4 Models to Save My Self-Improving Agent. All 4 Failed. AgentSelfEdit is an open-source sidecar that rewrites its own system prompt from execution feedback. It A/B tests edits and promotes only statistically-proven winners. Repo: github.com/deghosal-2026/agent-self-edit/tree/v0.3.0 Release notes: docs/release/v0.3.0/release-notes.md Field test report: docs/field-test/v0.3.0/FIELD_TEST_REPORT.md For a while I had a decent excuse for the optimizer looking weak. Maybe classification was just a bad place to judge it. Classification is rigid. Exact labels. Unforgiving scoring. Easy place for an LLM to look brittle. So when v0.3.0 got enough infrastructure behind it, I pushed beyond classification. I ran reduced-cost cheap-smoke loops on extraction, generation, and mixed-domain corpora. I expected some pain. I did not expect the same failure to show up three times wearing different clothes. The Cheap-Smoke Runs Were Small, But They Were Enough These were not full-blown multi-iteration field tests on every corpus. The report is careful about that, and I want to be too. The samples were smaller. The goal was narrower. I wanted to answer one question: is the analyzer's weakness classification-specific, or does it generalize? It generalizes. That answer showed up fast. Extraction Looked Reasonable Right Up Until It Didn't The extraction run was almost boring, which is part of what made it useful. Observed pattern: iteration 1: 0 up, 0 down iteration 2: 1 up, 0 down, effect_size = 0.0455 iteration 3: 1 up, 2 down The analyzer wasn't acting crazy. It kept proposing local tweaks around field naming and formatting: exact field names lowercase conventions concise formatting output consistency All of that sounds normal. That's what makes the result interesting. The analyzer can usually smell a local problem. It just keeps reaching for local instructions as if that alone will carry the day. Generation Was the One That Hurt My Ego a Little Generation exposed something classification only hinted at. The analyzer proposed edits that sounded smart to me when I read them: follow structure more strictly avoid generic content stick to format constraints That is the kind of advice prompt engineers say to each other all the time. Then I looked at the results: iteration 1: 1 improved, 7 regressed iteration 2: 0 proposals iteration 3: 1 improved, 2 regressed That was one of the most useful surprises in v0.3.0. Generation made it painfully obvious that an edit can sound more disciplined and still make the model worse. The prompt got tighter. The assistant got more obedient. The task results got uglier. I think a lot of us have done this by hand and talked ourselves into calling it improvement. This time the corpus didn't let me get away with that story. Mixed-Domain Was the Only One That Gave Me a Little Hope Mixed-domain was the most interesting non-classification run. Observed pattern: iteration 1: 0 up, 0 down iteration 2: 0 up, 0 down iteration 3: 2 up, 0 down, effect_size = inf, p = 0.46 That third iteration was the first non-classification cheap-smoke result that looked like a real little pocket of signal. But even there, the analyzer needed time to escape its bad habits. The first two iterations were still generic formatting and conciseness advice. Only later did something more domain-specific show up. So mixed-domain gave me two feelings at once: maybe broader corpora help, and no, broader corpora do not magically fix shallow search. The Pattern That Kept Coming Back Across classification, extraction, generation, and mixed-domain, the same structure kept repeating: the analyzer saw a plausible local issue it proposed a local wording tweak the tweak sometimes helped a little it usually didn't help enough sometimes it over-corrected and made things worse That is not four different failures. It is one failure pattern with different surface details. In classification it looked like urgency-boundary rewrites. In extraction it looked like field-format micro-edits. In generation it looked like stricter output discipline. In mixed-domain it started as generic clarity advice. Same habit. Different task. That was the big cross-domain learning in v0.3.0. Why This Surprised Me I expected some domains to be better or worse. What I did not expect was for the optimizer's personality to show up this clearly. It has a favorite move: see a plausible local error and patch it with a local wording rule. That is not nothing. It means the analyzer is not random. It is picking up real failure patterns. But it also means the current problem is not really about one corpus or one scorer. The deeper problem is that the analyzer does not explore broadly enough. Once I saw that across corpora, it got a lot harder to blame the wrong thing. What Worked I don't want to oversell the failure here. Some things did work. The framework could actually run across domains. That sounds basic, but earlier versions did not earn that assumption. v0.3.0 did. The analyzer was usually plausible. The issue was not total nonsense. The issue was narrowness. And generation gave me one of the sharpest developer lessons in the whole release: stricter prompt wording can absolutely make outputs worse while still sounding like an upgrade. What I Learned Cross-domain evidence is a reality check. A lot of AI system stories sound convincing inside one corpus. Once the same behavior starts showing up in extraction, generation, and mixed-domain runs, your favorite explanation gets a lot less special. I also learned, again, that "sounds right" is a useless quality bar for prompt edits. Generation made that painfully clear. Generation also taught me something specific I did not expect to value this much: it may be the best corpus for catching harmful "tighten the wording" edits. Classification is good at showing missed labels and weak boundaries. Generation is better at showing when stricter instructions make the system sound more obedient while actually making outputs worse. And I learned that the next real frontier here is search breadth. Not just a bigger model. Not just more runs. The analyzer has to get better at escaping its first obvious explanation of the failures. Why Developers Should Care If you're building systems that revise prompts from feedback, don't assume poor results in one domain are just a domain artifact. Run the same loop somewhere with a different failure surface. You might find what I found: the optimizer keeps making the same kind of move, regardless of the task. If that happens, the bug is not in the corpus. The habit is in the optimizer. I went into these cross-domain runs hoping they would make the optimizer look better. They did something more useful instead. They made the problem harder to misdiagnose. If you were designing the next corpus for a system like this, would you bias toward rigid tasks like classification, open-ended tasks like generation, or mixed-domain prompts that force broader search? And have you seen an optimizer in your own work keep making the same kind of "reasonable" edit over and over, even when the task changes?