A reader found the flaw in my code-graph test

I published a result that said my tool scored 34.5% and another scored 5%. A reader on Reddit asked two questions about how I counted. I went back, fixed the method, tested 40 more commits, and the result didn't hold. This is what went wrong, and how I'd run this kind of test now. The test I build EngramGraph, a code graph. I wanted to know if a code graph can tell an AI agent which other files need to change when a function changes. So I compared it with code-review-graph, an open source tool with about 32k stars on GitHub. The method was simple. Take a real pydantic commit. Go back to the parent commit and index it. Ask each tool who calls the changed functions. Compare the files it names with the files the commit really changed. I froze the method in a file before running it, and I ran 20 commits. That felt rigorous. It wasn't enough. What the reader asked The first question: when a tool declines to answer, is that a miss, or is it dropped from the denominator? It was a miss. An empty prediction scored 0 for that commit, and I averaged across commits. code-review-graph declined 18 of 42 lookups, so most of its score came from not answering. My headline was mostly measuring how often each tool answered, not whether its answers were right. The second question: what's the ceiling? Many files change together without any call between them, so 34.5% means something different against a ceiling of 40% than against 100%. I hadn't measured one. They also suggested showing per-commit numbers next to the pooled ones. I'm grateful for all three. None of this needed graph expertise, just someone reading the method carefully. What a decline really was I had described those 18 lookups as "returns nothing". Reading the source showed that was wrong. With a bare function name, code-review-graph searches names, paths and signatures as full text. Underscores count as word breaks, so rebuild_model_fields also matches test_rebuild_model_fields_preserves_description. When more than one node matches, it stops and returns a ranked candidate list, asking you to retry with a full name. That's a reasonable design for an agent. My script treated the reply as empty. What I changed I froze a new method, including every number I'd report. I used 40 new commits, chosen by the same rule, none from the first 20. I kept the same tool versions, so the rounds could be combined. I made the retry part of the method, not an analysis added afterwards. I re-ran the first commit from round one and got identical EngramGraph results, so the setup hadn't drifted. What happened My first run said one tool was far ahead. Across all 60 commits, with 171 files to find: Without a retry, EngramGraph found 29 files and code-review-graph found 7. With a retry, code-review-graph found 21, and 51% of its answers were right. EngramGraph's were right 35% of the time. On the 17 commits where both answered, each found 21 files, EngramGraph from 50 guesses and code-review-graph from 41. So the honest summary depends on whether your agent retries, and I haven't measured that. If it does, the two are about level and code-review-graph is more accurate. Calls also explain less than I assumed. Only about 30% of the co-changed files mention a changed function at all, so that's a rough ceiling for any tool working from calls. What I'd do differently Report how often a tool answered, separately from how often it was right. Never let an empty answer quietly count as a wrong one, or the reverse. Estimate a ceiling before reading a score. Show per-commit numbers. Round one's median was 0 for both tools, and the average hid that. Read the other tool's source before describing its behaviour. Treat a result from 20 commits as a question, not a finding. Don't compare your own product with someone else's and call it a finding. I set the method, chose the data and wrote the conclusion, so I wasn't a neutral judge. I've stopped using these tests for that kind of comparison. I still run them, because they're the best way I have to find what to fix. What I'm fixing The most useful thing that came out of this was a signal neither tool uses: files that historically change together. Added to EngramGraph's callers on the 40 new commits, it found 29 files instead of 19, at a cost of 6 points of accuracy. It's only been checked on one repository. The numbers All tables use the same scoring: files predicted, files correct, share correct when it answered, share of the files to find, and F1 pooled across commits. "Answered" counts commits with at least one prediction. Round one, as first published (20 commits, averaged per commit, empty answers score 0): Precision Recall F1 EngramGraph 0.345 0.345 0.337 code-review-graph, bare name 0.05 0.05 0.05 code-review-graph, retry (added after the run) 0.15 0.145 0.139 Round one, pooled (45 files to find): Answered Predicted Correct Right Found F1 EngramGraph 11 / 20 18 10 56% 22% 0.32 code-review-graph, bare name 1 / 20 1 1 100% 2% 0.04 code-review-graph, retry 4 / 20 8 5 63% 11% 0.19 Round two, frozen in advance (40 new commits, 126 files to find): Answered Predicted Correct Right Found F1 EngramGraph 24 / 40 65 19 29% 15% 0.20 code-review-graph, bare name 10 / 40 16 6 38% 5% 0.09 code-review-graph, retry 13 / 40 33 16 48% 13% 0.20 All 60 commits (171 files to find): Answered Predicted Correct Right Found F1 EngramGraph 35 / 60 83 29 35% 17% 0.23 code-review-graph, bare name 11 / 60 17 7 41% 4% 0.07 code-review-graph, retry 17 / 60 41 21 51% 12% 0.20 Git history, new 40 commits only (top 2 files per commit, using only commits before the parent): Predicted Correct Right Found F1 EngramGraph callers 65 19 29% 15% 0.20 Git history alone 80 16 20% 13% 0.16 Both 125 29 23% 23% 0.23 Retries in round two: code-review-graph declined 47 of 78 lookups. Retrying found caller files for 24, found no callers in other files for 8, and had no usable candidate for 15. My script made those retries, picking the candidate in the seed file. A real agent might not retry, or might pick wrong. Ceiling: 51 of 171 co-changed files mention a seed function by name, from a text search on the parent commit. That's an estimate. Limits: Python only, one repository, direct callers only. "Changed in the same commit" stands in for "affected", so a real caller that didn't need changing counts as wrong. Tool versions were code-review-graph 2.3.9 and EngramGraph 0.13.0. Its main feature, impact analysis from a diff, wasn't tested. In round one the pooled numbers and retry were added after the run; in round two everything was frozen first. I build EngramGraph, so weigh that. This article was written with the help of AI, based on my experiment notes and the scoring output.