Does Static Typing Reduce AI's Bugs? I Had Claude Code Build the Same Git Clone Three Times Each in TypeScript and JavaScript

"If an AI is writing your code, TypeScript is the safer choice." You hear it a lot, and I more or less believed it too. A type checker tells you what's wrong, so an AI should reach correct code more easily with one. But is that actually true? When an AI writes the code, does static typing (TypeScript) versus dynamic typing (JavaScript) really change how many bugs are left behind? This experiment was inspired by Yusuke Endoh's article here on DEV: Which Programming Language Is Best for Claude Code? Yusuke Endoh Yusuke Endoh Yusuke Endoh Follow Mar 5 Which Programming Language Is Best for Claude Code? #ai #llm #programming 50 reactions 13 comments 8 min read He had Claude Code write a simplified Git in many languages, 20 times each, and compared time and cost. TypeScript took about 1.6 times as long and cost about 1.6 times as much as JavaScript, and statically typed languages were slower and more expensive overall. The programs, though, were only a few hundred lines. He closed by suggesting that the benefits of static typing might show up at a larger scale, and left that for someone to test. So I scaled the task up to a few thousand lines and compared TypeScript and JavaScript again, this time including how many bugs were left. This is an English version of an article I first published in Japanese on Zenn. TL;DR At a few thousand lines, there was no clear difference between TypeScript and JavaScript. Put the other way around, the JavaScript advantage seen at a few hundred lines in Endoh's article went away. I had Claude Opus 5.5 build the same Git-compatible CLI from scratch, three times in TypeScript and three times in JavaScript. Here is what happened: Bugs left: 0 of 3 TypeScript runs and 1 of 3 JavaScript runs ended with a bug (one bug in total) Tokens: about 1M output tokens per run in both languages. Including input-side tokens, TypeScript used 16% more in total, but the ranges of the three runs overlap The one clear difference in the numbers was code size: TypeScript was about 20% longer (the type annotations) "0 vs 1" looks like a TypeScript win, but with three runs per side it can't be told apart from chance. Token usage varied by more than 10% even among the three TypeScript runs, and the gap between the languages fell inside that spread. That said, there was one interesting trend in how the AI worked. The JavaScript runs ran their unit tests about twice as often as the TypeScript runs. Without a type checker, the AI seems to have checked its work by running tests more often. At this scale it didn't cost anything noticeable, but at a larger scale it may become a real disadvantage for dynamic typing (more on this below). Mean of 3 runs TypeScript JavaScript Difference Runs with full marks on the hidden tests at the end 3 / 3 2 / 3 – Total tokens (all three versions) 67.7M 58.2M TS +16% — Output 1.00M 1.02M TS −2% — Cache reads 64.5M 55.2M TS +17% — Cache writes 2.13M 1.91M TS +12% — Uncached input 50K 28K TS +78% Time waiting for the model (all three versions) 9,801 s 9,844 s About the same Implementation size, final version 4,985 lines 4,153 lines TS +20% Cost (estimate, see note 1) $49.9 $46.7 TS +7% Everything is public: the materials, the specs, the tests and the AI's logs. Ikepeeee / ts-vs-js-agent-experiment TypeScript vs JavaScript: a controlled experiment AI agents implement the same specification (tgit v0.1.0 to v0.3.0) from scratch, once in TypeScript and once in JavaScript The experiment compares the bugs that remain and the cost of getting there. The only difference between the conditions is the language, along with the type checking that comes with it. Japanese version: README_ja.md. This English version is canonical; where the two differ, this one prevails. Layout Path Contents Read by README.md This file (Japanese: README_ja.md) Humans experiment-design.md The design of the controlled experiment. The run procedure is chapter 13 (Japanese: experiment-design_ja.md) Humans main.lask Every operation of the experiment, as lask tasks Humans spec/spec-v1.md to spec-v3.md The tgit specification (English). The same for both conditions AI (one version at a time) ts/AGENTS.md Instructions placed in the repository for the TypeScript condition AI js/AGENTS.md Instructions placed in the repository for the JavaScript condition … View on GitHub The setup The task: a Git-compatible CLI called "tgit" The AI built "tgit", a Git clone. It reads and writes repositories in the same format as the real Git, and its output has to match the real Git character for character. I chose Git because the right answer is unambiguous. Run the same operations on the real Git and compare the results, and correctness can be judged mechanically. The spec came in three versions. Each version was handed over in turn, and the AI kept building on the same repository. Version Main commands added Hidden tests (cumulative) v0.1.0 init, add, status, commit, log, branch, switch, and more 82 v0.2.0 diff, show, merge-base, merge 117 v0.3.0 tag, reset, cherry-pick, revert, reflog 151 Only the language differs The only difference between the conditions is the language and the type checking. TypeScript: strict and noUncheckedIndexedAccess on, no any, and passing tsc is part of "done" JavaScript: no type annotations (JSDoc types included) and no type checking The instructions given to the AI (AGENTS.md) are identical word for word, apart from those few lines. Both run directly on Node.js with no build step and no runtime dependencies. The AI was Claude Opus 5.5 in Claude Code, running inside a Docker container. It had the public tests and a tool that compares results against the real Git, so it could check its own work as it went. The hidden tests used for grading were not visible to it. A deliberately small context I turned off Claude Code's 1M-token context with CLAUDE_CODE_DISABLE_1M_CONTEXT=1, limiting it to 200K tokens. The idea was to make it easier to test one hypothesis: types start to matter once the code no longer fits in the AI's head. When the conversation reaches 200K tokens, Claude Code summarizes it automatically (compaction) and carries on. There were 27 compactions across the three TypeScript runs and 29 across the JavaScript runs, or 8 to 11 per run. The AI repeatedly summarized its own memory while it worked. Even so, the final code was a few thousand lines, still small enough to fit in 200K tokens on its own. What filled the context was the conversation, such as files it read and test output, rather than the code itself. How it was graded Besides the hidden tests, I ran 2,000 random sequences of operations per version and compared tgit with the real Git on: exit codes and output the contents of .git (objects, refs, the index) the files in the working tree whether the real Git's git fsck --strict accepts the repository tgit produced Matching output isn't enough: if .git differs behind the scenes, it fails. Two hidden tests and one random sequence turned out to be faulty on the test side during grading, and I excluded them (see note 2). Results: nearly perfect, with one bug left The first surprise was how accurately the AI built it in both languages. Every run got full marks on the public tests, and almost full marks on the hidden tests. Run Language v0.1.0 v0.2.0 v0.3.0 Random (2,000 per version) AN01 TS 82/82 117/117 151/151 All match AN02 TS 82/82 117/117 151/151 All match AN03 TS 82/82 117/117 151/151 All match BN01 JS 82/82 117/117 151/151 All match BN02 JS 82/82 117/117 151/151 All match BN03 JS 81/82 116/117 150/151 All match The only bug Only one bug survived. It was in BN03 (JavaScript): what happens when you add a file that .gitignore ignores. The real Git stops with an error and writes nothing. BN03 got the error exit code right, but wrote an empty .git/index before failing. The output was correct, so it would have gone unnoticed without comparing the contents of .git. It went in during the first version and was still there at the end. This isn't the kind of bug a type checker catches, though. It's a matter of ordering: writing a file before deciding there's an error. You can write exactly the same thing in TypeScript. Tokens: runs differ more than languages Next, token usage. The chart shows output tokens per run, stacked by version. TypeScript averaged about 1.00M and JavaScript about 1.02M: practically the same. What stands out more is the spread within each language. In JavaScript, the largest run (BN01) and the smallest (BN03) differ by almost 250K. v0.2.0, which adds merge, was the heaviest version in every run, at 50–60% of the total. Now the breakdown, including tokens other than output. Each type is indexed to the JavaScript mean. TypeScript used more of the input-side tokens: cache reads, cache writes and uncached input. For output and thinking, JavaScript actually used slightly more. For every type, though, the ranges of the three runs overlap. The numbers: Token type (all three versions) TypeScript mean (min–max) JavaScript mean (min–max) Difference in means Ranges Output 1.00M (0.95M–1.06M) 1.02M (0.89M–1.14M) TS −2% Overlap Cache reads 64.5M (56.3M–69.4M) 55.2M (49.8M–61.0M) TS +17% Overlap Cache writes 2.13M (1.94M–2.33M) 1.91M (1.84M–2.02M) TS +12% Overlap Uncached input 50K (26K–64K) 28K (23K–36K) TS +78% Overlap Total 67.7M (59.4M–72.8M) 58.2M (53.0M–63.9M) TS +16% Overlap Thinking (for reference, see note 3) 566K (513K–603K) 616K (524K–731K) TS −8% Overlap The biggest difference was in cache reads: TypeScript used about 17% more on average. Still, the lowest TypeScript run (AN03) used fewer than the highest JavaScript run (BN02), so the ranges overlap. Why TypeScript cost more despite less output The cost (see note 1) was $49.9 for TypeScript and $46.7 for JavaScript. JavaScript produced more output tokens, yet TypeScript cost more. The reason is that most of the cost comes from the input side, especially cache reads, not from output. Assuming relative prices for each token type, I split the cost into parts (see note 4): Cost breakdown (mean of 3 runs, estimated) TypeScript JavaScript TS − JS Output ~$16.1 ~$16.4 −$0.3 Cache reads ~$20.8 ~$17.8 +$3.0 Cache writes ~$13.7 ~$12.3 +$1.4 Uncached input ~$0.2 ~$0.1 +$0.1 Cost recorded in the logs $49.9 $46.7 +$3.2 Output is only about a third of the cost, so JavaScript's 2% more output is worth a few dozen cents. TypeScript's cache reads, on the other hand, were about 9.3M tokens higher, which alone accounts for about $3. An agent re-reads the whole conversation from the cache on every turn. TypeScript code is about 20% longer, so every file read or write puts more into the conversation. Uncached input, mostly tool results, was also about 1.8 times higher. The per-turn difference is small, but over hundreds of turns it adds up to 17%. No types, still correct The fun part came when I ran the JavaScript code through TypeScript's type checker after the fact (checkJs). The final versions had 832 to 1,022 type errors per run. Final version, mean of 3 runs TypeScript JavaScript Implementation lines 4,985 4,153 Test code lines 381 431 Comparison scenarios the AI wrote 125 130 Type errors when type-checked afterwards – 925 Seen through types, the code is full of holes. Yet it scored nearly full marks on the hidden tests. Being "loose" at the type level barely turned into actual bugs. Why no difference? I think it's because the AI could check itself against the real Git. In both languages, it wrote more than 120 comparison scenarios against the real Git on average, and kept finding and fixing mismatches. With verification that strong, the mistakes a type checker would catch get caught by the tests first, types or no types. Trends that aren't significant, but caught my eye With three runs per side, nothing here is statistically meaningful. Still, a few things stood out in the logs. The type checker barely had anything to do The TypeScript runs ran tsc 40 to 55 times each. Only 2 or 3 of those checks per run reported a type error. Across all three runs and all three versions, the AI saw just these eight type errors: Run Version Type error AN01 v0.2.0 Referenced a name that doesn't exist (filter) AN01 v0.3.0 Mismatched type argument on Buffer AN02 v0.1.0 Referenced a property branch that type Head doesn't have AN02 v0.2.0 Assigned a value that might be undefined AN02 v0.2.0 Passed a readonly array where a mutable one was expected AN03 v0.3.0 Referenced a name that doesn't exist (peelTags) AN03 v0.3.0 Referenced a name that doesn't exist (readStateHead) AN03 v0.3.0 Too few arguments About half are things like "name not found" or "too few arguments" that crash the moment the code runs; one test run would find them without a type checker. The only catch unique to type checking was the possible undefined (from noUncheckedIndexedAccess). The AI simply wrote very little code with type errors in the first place. The JavaScript runs ran unit tests about twice as often The TypeScript runs ran unit tests (npm test) 13 to 19 times each; the JavaScript runs, 28 to 36 times. The ranges don't overlap, and JavaScript ran them about twice as often. The comparison runs against the real Git were about the same: 135–200 for TypeScript and 161–207 for JavaScript. The JavaScript runs also wrote about 13% more test code. Without a type checker to lean on, they may have made up for it by running tests more often. I didn't find type-like checks in the tests themselves, though (such as checking argument types). This may be an early sign that dynamic typing gets worse as code grows. A type checker tells you in one pass how far a change reaches. Without types, the only way to find out is to run tests. As code grows and changes reach further, there are more tests to run, and each run takes longer. Here, twice the test runs made no difference in tokens, but at a larger scale this verification cost could start to hurt JavaScript. TypeScript code was always longer The shortest TypeScript run (AN03, 4,796 lines) was still longer than the longest JavaScript run (BN01, 4,642 lines). It's one of the few measures where the ranges separate cleanly. That's the type annotations and interface definitions, as expected. More thinking in JavaScript, more cache reads in TypeScript Thinking tokens were about 9% higher in JavaScript, and cache reads about 17% higher in TypeScript. You could read it as "without types, it thinks more; with types, it reads more code." Both ranges overlap, though, so this may well be chance. Caveats "No difference was found" isn't the same as "no difference exists." This experiment has limits. Few runs: three per language. Small differences can't be detected. More runs would be well worth it One task, one model: a CLI with an unambiguous right answer that the AI could check itself against. A task with a vaguer spec, or another model, might give different results Some tools given to the AI were outdated: I forgot to commit a fix to the random-test generator, so the AI got the old version. Both languages got the same one, so the comparison is fair, but the AI may have struggled with situations that didn't matter Six runs at once: my Mac's sleep and the load threw off wall-clock times, so I compared time spent waiting for the model instead. Token counts may also be 1–2% high Few rule violations: npm view, which was not allowed, ran twice and npx prettier once. Neither affected the implementation Summary On a task with a checkable right answer, Claude Opus 5.5 wrote about equally correct code with about equal token usage in TypeScript and JavaScript The one bug left wasn't a type error but an ordering problem, the kind a type checker can't prevent Even in TypeScript, the type checker found type errors only 2 or 3 times per run. The AI rarely writes type errors to begin with In Endoh's article, TypeScript was at a disadvantage in time and cost on small programs. Here, at a few thousand lines, the cost gap shrank to about 7%. Even so, there was no clear sign that types reduce bugs. The model (Claude Opus 4.6 in his article, Claude Opus 5.5 here) and the scale of the task both differ, so a direct comparison isn't possible. But at least at this scale, static typing was neither a big win nor a big loss. Turned around: just going from a few hundred lines to a few thousand made TypeScript's disadvantage in Endoh's article mostly disappear. If the gap shrinks as the scale grows, then somewhere further up it might flip, with TypeScript clearly ahead. The JavaScript runs running unit tests twice as often can be read as a hint in that direction. The same comparison at tens of thousands of lines, where the whole codebase no longer fits in the context, seems well worth trying. "Use types when an AI writes your code" may matter less than we think, at least at a few thousand lines with strong tests. That said, this was three runs per language on a single task. More runs, larger codebases, and tasks with vaguer specs are all worth trying. All the materials are public, so take a look if you're interested. Feel free to use the code. Notes Cost is the sum of the estimates that Claude Code records in its logs for each session (modelUsage in the result record). It isn't an actual bill. I ran everything on a subscription (logged in to Claude Code), so these amounts weren't charged as such. Excluded tests: two hidden tests required a feature the spec doesn't mention, and one random sequence couldn't be compared because of how the real Git behaves. Without the exclusion, one TypeScript run and one JavaScript run would have had full marks on the hidden tests at the end. Thinking tokens are recorded in Claude Code's logs as a separate item from output. The logs don't show whether they're part of output, so they aren't in the total. The cost breakdown is an estimate. Taking the uncached input price as 1, I assumed cache reads at 0.1, cache writes at 2 (1-hour cache) and output at 5, found the base price that best fit the logged costs, and split the cost accordingly. It isn't based on official prices, and the fit is off by up to about 9% per run, so the parts don't add up exactly to the logged cost.