Humanity's Last Exam on Hugging Face: At Least 23 of the Top 24 Rows Use Tools, and the Board Has No Column for It

TL;DR On November 1, 2026 the Hugging Face board for Humanity's Last Exam (HLE) lists 96 rows from 64 model repos, 0 marked verified. Row 1 is 63.9, row 2 is 62.5, row 3 is 61.0. The board has no field for "tools allowed". The only signal is the free-text notes string. By my count 34 rows say "with tools", 12 say "no tools", and 50 say nothing. 21 of the top 24 rows say "with tools". The other 3 are unlabeled, and for two of them (Kimi-K3 at 56.0, Inkling at 46.0) the model card shows the number is the with-tools score. The first row that says "no tools" is rank 25. Where the same model has both a with-tools and a no-tools row, tools add 7.8 to 15.1 points (median 11.3, seven pairs). That is about four times the sampling noise of a 2,500-question test, and it is not a constant you can subtract. Among closed-book numbers the top six sit within about 2 points of each other, so their order is not established by this data. What is on the HLE board, and what does a row actually claim? HLE is a 2,500-question exam written by subject experts. The paper describes it as multimodal: roughly 14% of the questions need images, and about 24% are multiple choice, the rest short exact-match answers. Hugging Face lists the dataset as an official benchmark, so model repos can publish a score in a .eval_results/*.yaml file and the board collects them. A row is therefore a self-report. In my last post on this series I walked through how those YAML files become a board; here the relevant part is small. A row has a value, a source link, a pull request or commit, and an optional notes string. There is no structured field for tool access, test subset, judge model or number of runs. Two rows hint at what the board's default is. One submitter writes that their number is "with tools (full set)... not the no-tools default". Another says theirs is "evaluated with tool/browsing access enabled, not the closed-book default". I could not read the dataset's eval.yaml (it is gated), so I cannot confirm the default from the source, but if those two notes are right, the top of the board is made of off-default runs. How many rows say whether tools were used? I classified every row by its note: "with tools", "w/ tool" or "with tool" means tools; "no tools" or "without tools" means closed-book; anything else is unlabeled. Condition (from note text) Rows Share of 96 With tools 34 35% No tools 12 13% Unlabeled 50 52% Half the board does not say. That is not a criticism of any single submitter: the YAML format never asked. Ten rows also say "text-only", which means they were scored on the subset without images. The Artificial Analysis HLE page uses 2,158 text-only questions out of the 2,500, so a text-only row and a full-set row are scored on different tests. Three rows state a judge: two say GPT-4o (one adds that the task default is o3-mini) and one says GPT-5.6-luna. The other 93 rows say nothing about the grader. Why does the top of the board look the way it does? Sorted by value, the top 24 rows break down like this: 21 say "with tools", 3 are unlabeled, none say "no tools". The first explicit no-tools row, Ornith-1.5-397B at 44.6, is rank 25. The three unlabeled rows in the top 24 are Kimi-K3 (rank 5, 56.0), Hy3 (rank 10, 53.2) and Inkling (rank 23, 46.0). I opened the model cards: Kimi-K3. The card's HLE-Full row reads "43.5 / 56.0", and the footnote says each cell reports the score "without and with tool augmentation ... in that order". The board's 56.0 is the with-tools number. Its hle.yaml contains a value, a source and nothing else. The closed-book 43.5 is on the card but not on the board. With 43.5 it would sit near rank 27. Inkling. The card lists "HLE (text only) 29.7%" and "HLE (with tools) 46.0%". The board's 46.0 is the with-tools figure. The text-only figure appears nowhere on the board. Hy3. I found no HLE line in the card text, so I cannot resolve this row. I count it as unknown. So the accurate statement is "at least 23 of the top 24". Row 1 is also worth a look: its note reads "With tools; harness not specified in the model card." The top score comes with the tool set and the harness both unspecified. How big is the tool uplift, and can you correct for it? Several repos publish both numbers, so I paired rows that belong to the same model. Seven pairs have both rows explicitly labeled. Six more come from gpt-oss, where the paired rows share a reasoning level and the no-tools rows carry no tools label (the pairing follows the cited OpenAI paper, arXiv 2508.10925, so treat it as weaker). Model With tools No tools Uplift Beam-1.0-501B 51.1 36.0 +15.1 Hy4-preview (text-only) 55.4 43.4 +12.0 Ornith-1.5-397B 56.1 44.6 +11.5 Nemotron-3-Ultra NVFP4 37.4 26.1 +11.3 Nemotron-3-Ultra BF16 37.4 26.7 +10.7 Ornith-1.5-9B 30.5 20.2 +10.3 Ornith-1.5-35B 33.4 25.6 +7.8 gpt-oss-20b (high) 17.3 10.9 +6.4 gpt-oss-120b (high) 19.0 14.9 +4.1 gpt-oss-120b (low) 9.1 5.2 +3.9 gpt-oss-120b (medium) 11.3 8.6 +2.7 gpt-oss-20b (low) 6.3 4.2 +2.1 gpt-oss-20b (medium) 8.8 7.0 +1.8 For the seven explicit pairs the median uplift is 11.3 points with a range of 7.8 to 15.1. Two of those are the same Nemotron model in two precisions, so call it six independent pairs. Across all 13 pairs the range is 1.8 to 15.1, an eightfold spread. That spread is the practical point. There is no single "tools constant" to subtract from a with-tools row to get its closed-book equivalent. The uplift depends on the model, its training for tool use, the tool set and the harness. A with-tools row cannot be turned into a closed-book row by arithmetic; it can only be replaced by a closed-book measurement. For scale, the sampling noise of the test itself is small next to this. With n = 2,500 and accuracy near 45%, the standard error is 0.99 points and a 95% interval is about plus or minus 1.95. Two independent runs need to differ by about 2.8 points (3.0 on the 2,158-question subset) before the gap clears noise. Three of the gpt-oss uplifts (1.8, 2.1, 2.7) fall under that line, though the pairs are scored on the same questions, which lowers the real threshold by an amount I cannot compute without per-question results. Is the closed-book ordering any better? Take the closed-book numbers: the explicit no-tools rows plus Kimi-K3's 43.5 from its card. The top six run from 44.6 down to 42.6, a spread of 2.0 points. Every interval overlaps every other. Three of the six are text-only rows scored on 2,158 questions, so their intervals are slightly wider than drawn, and the rows without a subset note may or may not be on the same test. The honest reading is a cluster, not a ranking. This matches what I found on the GPQA Diamond board: when the test is small relative to the model gaps, listing models in order of value implies precision that is not there. How do I check the conditions on any board myself? The board JSON endpoint is undocumented and may change, so treat this as a snapshot tool. It reproduces the counts above. import json, re, collections, urllib.request URL = "https://huggingface.co/api/datasets/cais/hle/leaderboard" # undocumented, may change rows = json.load(urllib.request.urlopen( urllib.request.Request(URL, headers={"User-Agent": "Mozilla/5.0"}), timeout=60)) def condition(note): note = (note or "").lower() if re.search(r"with tools?|w/ tool", note): return "tools" if re.search(r"no tools?|without tools?", note): return "no_tools" return "unlabeled" for r in rows: r["cond"] = condition(r.get("notes")) print(len(rows), "rows;", collections.Counter(r["cond"] for r in rows)) print("top 24 by condition:", collections.Counter(r["cond"] for r in rows[:24])) by_model = collections.defaultdict(list) for r in rows: by_model[r["modelId"]].append(r) for model, rs in by_model.items(): t = [r for r in rs if r["cond"] == "tools"] n = [r for r in rs if r["cond"] == "no_tools"] if len(t) == 1 and len(n) == 1: print(f"{model}: {t[0]['value']} vs {n[0]['value']} (+{t[0]['value'] - n[0]['value']:.1f})") On November 1, 2026 it prints: 96 rows; Counter({'unlabeled': 50, 'tools': 34, 'no_tools': 12}) top 24 by condition: Counter({'tools': 21, 'unlabeled': 3}) ornith-ai/Ornith-1.5-397B: 56.1 vs 44.6 (+11.5) tencent/Hy4-preview: 55.4 vs 43.4 (+12.0) reflection/Beam-1.0-501B-A23B: 51.1 vs 36 (+15.1) nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16: 37.4 vs 26.7 (+10.7) nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4: 37.4 vs 26.1 (+11.3) ornith-ai/Ornith-1.5-35B-A3B: 33.4 vs 25.6 (+7.8) ornith-ai/Ornith-1.5-9B: 30.5 vs 20.2 (+10.3) The classifier reads free text, so it is a lower bound on labeled rows: a note phrased in some other way lands in "unlabeled". That is the same limitation a reader faces. What should a reader do with a board like this? Split by condition before you rank. With-tools, no-tools and unlabeled rows are three different lists. Treat unlabeled as unknown, not as the default. Open the model card. In two of three top-24 cases here, the card settled it in under a minute. Compare within one subset. Text-only (2,158 questions) and full-set (2,500) rows are different tests. Apply a noise floor of about 3 points between independent runs, and more if the judge differs. Look for the closed-book number the card reports but the board omits. Kimi-K3's 43.5 is a case where the card has more than the board. Ask submitters for three fields: tools used (yes or no, and which), subset, and judge. A board that stored those as structured fields would make the filter above a dropdown instead of a regex. FAQ Do tools make an HLE score invalid? No. A with-tools score is a legitimate measurement of a model plus its tool harness. It is just not comparable to a closed-book score, and the board puts them in one sorted list. Is the 63.9 at the top wrong? Nothing here shows it is wrong. It is self-reported, marked with tools, and its note says the harness is not specified, so it cannot be compared with rows run under a different harness. How many HLE questions does the board's test have? The dataset has 2,500 questions. Some rows are scored on the text-only subset, which Artificial Analysis puts at 2,158 questions, so the denominator differs between rows. Can I convert a with-tools score to a closed-book estimate? Not reliably. In the paired rows the uplift runs from 1.8 to 15.1 points, depending on the model and reasoning level. Why do unlabeled rows matter more than labeled ones? They are 50 of 96 rows, and the two I could resolve at the top of the board turned out to be with-tools numbers. Further reading GPQA Diamond Has 198 Questions: What the Hugging Face Leaderboard's 115 Rows Can and Cannot Rank When a 0.4-Point Lead Means Nothing: Reading Open-Model Leaderboard Margins Like a Statistician Sources HLE dataset and board: https://huggingface.co/datasets/cais/hle Humanity's Last Exam paper: https://arxiv.org/abs/2501.14249 Artificial Analysis HLE evaluation page (2,158 text-only questions): https://artificialanalysis.ai/evaluations/humanitys-last-exam Kimi-K3 model card: https://huggingface.co/moonshotai/Kimi-K3 Inkling model card: https://huggingface.co/thinkingmachines/Inkling gpt-oss paper: https://arxiv.org/abs/2508.10925 All counts in this article are my own measurements from the board JSON and the cited cards on November 1, 2026 (KST); I have no affiliation with any model, lab or benchmark named here. The classification of rows is mine, based on free-text notes, and the board endpoint is undocumented and may change.