Claude for business: testing three Claude models as SQL agents on a CRM

Claude for business decisions often come down to which model to put behind an agent, and this post shows how to settle that with a small test you can rebuild on your own data: an invented CRM with known traps, ground truth computed two ways, an agent loop with one read-only SQL tool, a scorer with a tolerance, and a cost per right answer instead of a cost per token. By the end you will be able to run the same comparison on your own questions and see what a right answer costs from each model. The trigger was the launch of [Claude Fable 5.2](https://www.anthropic.com/claude-fable-and-mythos-5-2) on 9 November, at $8 / $40 per million input and output tokens, twice [the list price](https://platform.claude.com/docs/en/about-claude/pricing) of Opus 5.5. I had Claude Code build the test, and a GitHub workflow made the API calls. Everything below is from the real files, and every code block is trimmed from them.  ## The setup in one paragraph Three models (Claude Fable 5.2, Claude Opus 5.5, Claude Sonnet 5.5), the same 8 questions from an invented sales director, effort medium\, 2 runs per question per condition, and two conditions: with the export's column documentation and without it. That is 32 runs per model and 96 in all. Every call, every query and every answer is written to a JSONL file. The whole thing cost $9.58 at list price. ## Building a CRM with traps on purpose pack.py\ generates an invented Danish software firm, Saltholm Data ApS, deterministically, so the same seed always gives the same database. It has 150 live deals plus 12 archived copies left over from merged duplicates, amounts in 4 currencies, 9 deals with two CFOs in the contacts table, and 16 live deals that entered proposal more than once. Each of those is there because it makes a naive total wrong. The schema the models see, trimmed to the table that matters most: \sql CREATE TABLE deals (  id INTEGER PRIMARY KEY,  company\_id INTEGER REFERENCES companies(id),  owner\_id INTEGER REFERENCES users(id),  name TEXT,  pipeline TEXT,         -- 'new\_business' or 'renewal'  stage TEXT,            -- current stage: lead, qualified, proposal, negotiation, closed\_won, closed\_lost  amount REAL,           -- deal value in the deal's own currency  currency TEXT,         -- DKK, EUR, SEK or NOK  close\_date TEXT,       -- expected close date for open deals, actual close date for closed deals  archived INTEGER,      -- 1 = merged into another deal and kept for history only  merged\_into INTEGER    -- the deal this one was merged into, when archived = 1 ); \\ The other tables are deal\_contacts\ (deal, contact, role), deal\_line\_items\, stage\_history\ (deal, stage, entered_at), fx\_rates\ and users\. In the "without documentation" condition, the comments after --\ are stripped, so the model sees bare column names. ## Ground truth, computed twice truth.py\ computes the answer to every question twice: once in plain Python over the generated records, and once in SQL over the SQLite file. The script exits 1 if the two disagree by more than half a krone. For the CFO question, the SQL uses EXISTS\ so a deal with two CFOs counts once: \sql select sum(d.amount\*f.dkk\_per\_unit) from deals d join fx\_rates f on f.currency=d.currency where d.archived=0 and d.stage in ('lead','qualified','proposal','negotiation') and exists (select 1 from deal\_contacts c where c.deal\_id=d.id and c.role='CFO') \\ It also computes the naive answers, so you know what each trap would cost: DKK 12,527,857 instead of DKK 10,983,557 for the open pipeline without currency conversion (14% too big), DKK 11,772,774 with the archived copies counted (7%), and DKK 6,342,447 instead of DKK 5,083,337 when you join to contacts and sum (25%).  ## The agent loop api\_job.py\ gives each model two tools: run\_sql\, which runs one SELECT against a read-only copy and returns at most 40 rows, and submit\_answer\, which takes the number, the SQL that produced it and a short explanation. The tool descriptions below are translated; the originals are in Danish, as is the question. \python TOOLS = \[    {"name": "run\_sql",     "description": "Run one SELECT against the CRM database (SQLite, read-only). "                    f"Returns at most {MAX\_ROWS} rows.",     "input\_schema": {"type": "object", "properties": {"sql": {"type": "string"}}, "required": \["sql"\]}},    {"name": "submit\_answer",     "description": "Submit your final answer: the number, the query that gave it and a short explanation.",     "input\_schema": {"type": "object", "properties": {         "value": {"type": "number"}, "sql": {"type": "string"}, "forklaring": {"type": "string"}},         "required": \["value", "sql", "forklaring"\]}}, \] \\ The loop is the plain tool-use loop: call the model, run every run\_sql\ it asked for, send the results back, and stop when submit\_answer\ arrives or after 14 turns. Errors are recorded, never hidden. Every one of the 96 runs ended with an answer. The connection opens the database with mode=ro\, so a model that tries an UPDATE gets an error back instead of a changed answer key. ## Scoring with a tolerance score.py\ counts an answer as right when it is within half a percent of the truth: \python right = abs(val - t) / abs(t)