AI benchmarks on Heroes III
Which model masters Might & Magic?
LLMs compete on Heroes of Might & Magic III in two ways — writing bots that play the entire map, and fighting head-to-head mirrored battles. Pick a discipline:
Latest league · S9 + DeepSeek · Complete
Who writes the best bot?
Ten LLMs. Seventeen bots. Each authoring run gets one isolated agentic session, a 300-game practice budget and the game film — a searchable log of its bot's decisions. The submitted bots now compete in 5,436 scored league games from 5,440 attempts on eight sealed held-out maps.
DeepSeek-V4-Pro-0813 joins with one completed authoring run at native max reasoning: A-only, N=1. Run B is not included in this result. Flash retains two independent max runs; the 14 original S9 bots remain frozen from their xhigh sessions. These provider-specific settings do not imply equal compute.
The main board includes DeepSeek. It combines 4,800 unchanged S9 + Flash games with 636 newly scored games from 640 new attempts. Expanded-field medians and ranks are recalculated against all 17 bots.
This extension awards no new title. Pro has one submitted run (A-only, N=1); its missing B is not scored as zero. The 16-bot Flash result remains available. Historical S9 standings and the S9 title playoff remain unchanged in the archive. Opus and Fable's second legs were skipped due to temporary technical issues; missing runs are not scored as zero.
The frozen regular-league gate permits up to 1% engine hangs across all attempts. This completed schedule has 4 failed attempts, including 4 hangs: 4/5,440 overall (0.07%) and 4/640 new attempts (0.63%). All hangs are in the new Pro A matchups; the 4,800 preserved games have none. Failed attempts remain excluded from scores and are never replayed. This is separate from the historical title playoff's zero-failure gate.
Latest standings · 10 models · 17 runs
Model scores · independent-run counts shown
Per game, a bot scores the peak-army margin (log-ratio of day-capped peak army vs the opponent). The schedule mirrors both seats; failed games are excluded and never replayed. A run's score is its median over its valid league games; a model's displayed score is the mean of its available runs. Pro, Opus and Fable have N=1. Active is session time actually spent, quota pauses excluded.
Scroll horizontally to compare all run columns →
# Writer model Score mean available runs Run A median margin [95% CI] Run B median margin [95% CI] Gap |A − B| Elims for / against Active A / B
1 Claude Opus 5.5 Frozen S9 · claude:claude-opus-5-5 · xhigh · N=1 +0.511 +0.511[+0.413, +0.603]Run rank #2 / 17640 scored / 640 attempts Skippedtemporary technical issues — 21 / 6 3.6 h —
2 GPT-6-Astra Frozen S9 · codex:gpt-6-astra · xhigh · N=2 +0.511 +0.531[+0.438, +0.609]Run rank #1 / 17636 scored / 640 attempts +0.491[+0.437, +0.565]Run rank #3 / 17640 scored / 640 attempts 0.04 97 / 5 2.5 h 2.8 h
3 GPT-6.1-Sol Frozen S9 · codex:gpt-6.1-sol · xhigh · N=2 +0.436 +0.473[+0.403, +0.542]Run rank #4 / 17640 scored / 640 attempts +0.399[+0.317, +0.483]Run rank #5 / 17640 scored / 640 attempts 0.07 78 / 5 4.3 h 2.1 h
4 Claude Sonnet 5.5 Frozen S9 · claude:claude-sonnet-5-5 · xhigh · N=2 +0.238 +0.289[+0.212, +0.376]Run rank #6 / 17640 scored / 640 attempts +0.188[+0.124, +0.278]Run rank #8 / 17640 scored / 640 attempts 0.10 78 / 0 6.3 h 2.8 h
5 Claude Fable 5.1 Frozen S9 · claude:claude-fable-5-1 · xhigh · N=1 +0.221 +0.221[+0.135, +0.291]Run rank #7 / 17640 scored / 640 attempts Skippedtemporary technical issues — 76 / 0 3.4 h —
6 GPT-5.6-Sol Frozen S9 · codex:gpt-5.6-sol · xhigh · N=2 +0.021 +0.003[-0.060, +0.066]Run rank #10 / 17640 scored / 640 attempts +0.040[-0.026, +0.105]Run rank #9 / 17640 scored / 640 attempts 0.04 54 / 32 3.5 h 2.8 h
7 DeepSeek-V4-Pro-0813 New · deepseek:deepseek-v4-pro · max · N=1 · A-only -0.220 -0.220[-0.287, -0.139]Run rank #11 / 17636 scored / 640 attempts Not includedA-only evaluation · N=1 — 7 / 36 3.0 h —
8 DeepSeek-V4.1-Flash-1017 Frozen Flash · deepseek:deepseek-flash · max · N=2 -0.375 -0.307[-0.383, -0.230]Run rank #12 / 17640 scored / 640 attempts -0.444[-0.568, -0.324]Run rank #13 / 17640 scored / 640 attempts 0.14 118 / 89 2.6 h 2.1 h
9 GPT-6-Luna Frozen S9 · codex:gpt-6-luna · xhigh · N=2 -0.916 -0.841[-0.924, -0.754]Run rank #14 / 17640 scored / 640 attempts -0.991[-1.029, -0.935]Run rank #15 / 17640 scored / 640 attempts 0.15 35 / 58 1.9 h 1.6 h
10 Claude Haiku 4.5 Frozen S9 · claude:claude-haiku-4-5 · xhigh · N=2 -2.782 -3.457[-3.549, -3.400]Run rank #17 / 17640 scored / 640 attempts -2.106[-2.159, -2.033]Run rank #16 / 17640 scored / 640 attempts 1.35 0 / 333 0.5 h 1.2 h
Head-to-head
Every run against every run
Seed-paired median margin of the row bot against the column bot. Each matchup schedules 20 seeds from both seats (40 attempts). Only complete seed pairs enter this paired statistic and its 95% CI; valid unpaired games still count in run scores. Green: row ahead; red: row behind. Select a cell for its sample size, exclusions and interval.
| row vs column | Astra A | Opus 5.5 A | Astra B | 6.1 Sol A | 6.1 Sol B | Sonnet 5.5 A | Fable 5.1 A | Sonnet 5.5 B | 5.6 Sol B | 5.6 Sol A | DeepSeek Pro 0813 A | DeepSeek 1017 A | DeepSeek 1017 B | 6 Luna A | 6 Luna B | Haiku 4.5 B | Haiku 4.5 A |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Astra A +0.53 | -0.195% CI [-0.205, +0.117]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.195% CI [-0.178, +0.041]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.195% CI [-0.078, +0.290]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.295% CI [+0.045, +0.214]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.295% CI [+0.047, +0.332]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.495% CI [+0.213, +0.548]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.295% CI [+0.175, +0.470]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.395% CI [+0.134, +0.392]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.595% CI [+0.325, +0.604]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.695% CI [+0.492, +0.744]17 complete seed pairs; 36 scored / 40 attempts; 4 failed, including 4 hangs | +0.895% CI [+0.677, +0.870]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +1.195% CI [+0.892, +1.219]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +1.395% CI [+0.921, +1.531]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +1.595% CI [+1.299, +1.540]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +2.495% CI [+2.199, +2.551]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +4.195% CI [+3.364, +4.450]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | |
| Opus 5.5 A +0.51 | +0.195% CI [-0.117, +0.205]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.095% CI [-0.203, +0.186]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.095% CI [-0.217, +0.150]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.095% CI [-0.134, +0.130]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.195% CI [-0.026, +0.184]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.295% CI [+0.086, +0.320]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.395% CI [+0.173, +0.381]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.495% CI [+0.174, +0.591]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.495% CI [+0.118, +0.555]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.595% CI [+0.413, +0.660]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.895% CI [+0.644, +1.149]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +1.395% CI [+1.065, +1.372]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +1.195% CI [+1.003, +1.403]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +1.495% CI [+1.278, +1.619]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +2.695% CI [+2.404, +2.840]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +4.595% CI [+3.828, +5.032]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | |
| Astra B +0.49 | +0.195% CI [-0.041, +0.178]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.095% CI [-0.186, +0.203]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.195% CI [+0.036, +0.265]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.195% CI [+0.027, +0.276]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.395% CI [+0.141, +0.441]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.295% CI [+0.122, +0.463]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.395% CI [+0.195, +0.395]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.395% CI [+0.161, +0.440]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.595% CI [+0.319, +0.561]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.595% CI [+0.492, +0.691]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.695% CI [+0.363, +0.891]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.895% CI [+0.573, +0.939]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +1.295% CI [+1.022, +1.413]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +1.295% CI [+1.104, +1.488]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +2.695% CI [+2.384, +2.806]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +4.095% CI [+3.623, +4.633]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | |
| 6.1 Sol A +0.47 | -0.195% CI [-0.290, +0.078]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.095% CI [-0.150, +0.217]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.195% CI [-0.265, -0.036]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.195% CI [-0.212, +0.042]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.195% CI [-0.117, +0.233]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.395% CI [+0.217, +0.525]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.195% CI [+0.044, +0.283]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.495% CI [+0.237, +0.505]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.395% CI [+0.200, +0.357]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.595% CI [+0.314, +0.624]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.995% CI [+0.702, +1.141]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.895% CI [+0.719, +1.091]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +1.495% CI [+1.024, +1.582]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +1.295% CI [+1.100, +1.472]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +2.395% CI [+2.127, +2.416]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +3.495% CI [+3.292, +4.097]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | |
| 6.1 Sol B +0.40 | -0.295% CI [-0.214, -0.045]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.095% CI [-0.130, +0.134]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.195% CI [-0.276, -0.027]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.195% CI [-0.042, +0.212]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.095% CI [-0.018, +0.243]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.295% CI [-0.036, +0.263]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.195% CI [-0.044, +0.211]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.295% CI [+0.106, +0.306]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.595% CI [+0.258, +0.548]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.595% CI [+0.461, +0.643]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.795% CI [+0.470, +0.874]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.895% CI [+0.645, +1.122]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +1.395% CI [+0.978, +1.363]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +1.195% CI [+1.034, +1.448]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +2.395% CI [+2.090, +2.441]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +3.895% CI [+3.477, +4.901]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | |
| Sonnet 5.5 A +0.29 | -0.295% CI [-0.332, -0.047]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.195% CI [-0.184, +0.026]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.395% CI [-0.441, -0.141]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.195% CI [-0.233, +0.117]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.095% CI [-0.243, +0.018]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.295% CI [-0.040, +0.297]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.195% CI [-0.214, +0.206]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.195% CI [+0.013, +0.155]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.495% CI [+0.044, +0.485]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.495% CI [+0.247, +0.562]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.595% CI [+0.412, +0.832]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.795% CI [+0.594, +0.956]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +1.195% CI [+0.775, +1.587]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +1.195% CI [+0.957, +1.194]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +2.495% CI [+1.958, +2.562]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +3.795% CI [+3.157, +4.073]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | |
| Fable 5.1 A +0.22 | -0.495% CI [-0.548, -0.213]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.295% CI [-0.320, -0.086]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.295% CI [-0.463, -0.122]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.395% CI [-0.525, -0.217]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.295% CI [-0.263, +0.036]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.295% CI [-0.297, +0.040]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.195% CI [-0.008, +0.169]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.395% CI [+0.155, +0.361]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.395% CI [+0.113, +0.318]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.695% CI [+0.335, +0.864]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.595% CI [+0.333, +0.557]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +1.095% CI [+0.617, +1.431]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.995% CI [+0.628, +1.179]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +1.195% CI [+0.913, +1.367]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +2.395% CI [+2.190, +2.422]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +3.295% CI [+3.022, +3.442]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | |
| Sonnet 5.5 B +0.19 | -0.295% CI [-0.470, -0.175]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.395% CI [-0.381, -0.173]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.395% CI [-0.395, -0.195]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.195% CI [-0.283, -0.044]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.195% CI [-0.211, +0.044]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.195% CI [-0.206, +0.214]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.195% CI [-0.169, +0.008]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.195% CI [-0.005, +0.270]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.295% CI [+0.095, +0.277]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.595% CI [+0.232, +0.686]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.495% CI [+0.272, +0.580]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.695% CI [+0.483, +0.820]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +1.195% CI [+0.920, +1.403]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +1.395% CI [+1.136, +1.504]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +2.695% CI [+2.458, +2.766]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +4.195% CI [+3.545, +4.519]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | |
| 5.6 Sol B +0.04 | -0.395% CI [-0.392, -0.134]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.495% CI [-0.591, -0.174]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.395% CI [-0.440, -0.161]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.495% CI [-0.505, -0.237]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.295% CI [-0.306, -0.106]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.195% CI [-0.155, -0.013]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.395% CI [-0.361, -0.155]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.195% CI [-0.270, +0.005]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.095% CI [-0.178, +0.261]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.495% CI [+0.191, +0.460]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.395% CI [+0.176, +0.501]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.495% CI [+0.265, +0.679]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.895% CI [+0.550, +0.940]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.895% CI [+0.628, +0.954]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +2.295% CI [+2.098, +2.313]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +3.895% CI [+2.753, +4.347]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | |
| 5.6 Sol A +0.00 | -0.595% CI [-0.604, -0.325]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.495% CI [-0.555, -0.118]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.595% CI [-0.561, -0.319]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.395% CI [-0.357, -0.200]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.595% CI [-0.548, -0.258]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.495% CI [-0.485, -0.044]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.395% CI [-0.318, -0.113]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.295% CI [-0.277, -0.095]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.095% CI [-0.261, +0.178]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.295% CI [-0.033, +0.382]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.395% CI [+0.104, +0.411]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.795% CI [+0.260, +1.005]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +1.295% CI [+0.665, +1.421]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +1.195% CI [+0.969, +1.226]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +2.195% CI [+1.931, +2.285]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +4.095% CI [+3.267, +4.611]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | |
| DeepSeek Pro 0813 A -0.22 | -0.695% CI [-0.744, -0.492]17 complete seed pairs; 36 scored / 40 attempts; 4 failed, including 4 hangs | -0.595% CI [-0.660, -0.413]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.595% CI [-0.691, -0.492]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.595% CI [-0.624, -0.314]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.595% CI [-0.643, -0.461]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.495% CI [-0.562, -0.247]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.695% CI [-0.864, -0.335]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.595% CI [-0.686, -0.232]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.495% CI [-0.460, -0.191]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.295% CI [-0.382, +0.033]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.295% CI [-0.035, +0.384]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.095% CI [-0.158, +0.180]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.595% CI [+0.108, +0.605]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.795% CI [+0.654, +0.903]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +2.095% CI [+1.891, +2.209]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +3.395% CI [+2.857, +4.308]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | |
| DeepSeek 1017 A -0.31 | -0.895% CI [-0.870, -0.677]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.895% CI [-1.149, -0.644]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.695% CI [-0.891, -0.363]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.995% CI [-1.141, -0.702]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.795% CI [-0.874, -0.470]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.595% CI [-0.832, -0.412]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.595% CI [-0.557, -0.333]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.495% CI [-0.580, -0.272]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.395% CI [-0.501, -0.176]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.395% CI [-0.411, -0.104]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.295% CI [-0.384, +0.035]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.395% CI [-0.438, -0.079]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.495% CI [+0.094, +0.561]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.895% CI [+0.600, +0.917]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +1.995% CI [+1.683, +2.081]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +3.195% CI [+2.922, +3.906]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | |
| DeepSeek 1017 B -0.44 | -1.195% CI [-1.219, -0.892]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -1.395% CI [-1.372, -1.065]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.895% CI [-0.939, -0.573]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.895% CI [-1.091, -0.719]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.895% CI [-1.122, -0.645]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.795% CI [-0.956, -0.594]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -1.095% CI [-1.431, -0.617]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.695% CI [-0.820, -0.483]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.495% CI [-0.679, -0.265]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.795% CI [-1.005, -0.260]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.095% CI [-0.180, +0.158]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.395% CI [+0.079, +0.438]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.895% CI [+0.050, +1.008]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.895% CI [+0.692, +0.974]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +1.895% CI [+1.572, +1.956]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +3.495% CI [+2.453, +3.784]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | |
| 6 Luna A -0.84 | -1.395% CI [-1.531, -0.921]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -1.195% CI [-1.403, -1.003]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -1.295% CI [-1.413, -1.022]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -1.495% CI [-1.582, -1.024]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -1.395% CI [-1.363, -0.978]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -1.195% CI [-1.587, -0.775]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.995% CI [-1.179, -0.628]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -1.195% CI [-1.403, -0.920]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.895% CI [-0.940, -0.550]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -1.295% CI [-1.421, -0.665]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.595% CI [-0.605, -0.108]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.495% CI [-0.561, -0.094]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.895% CI [-1.008, -0.050]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +0.295% CI [-0.132, +0.450]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +1.595% CI [+1.131, +1.666]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +3.295% CI [+2.388, +3.818]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | |
| 6 Luna B -0.99 | -1.595% CI [-1.540, -1.299]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -1.495% CI [-1.619, -1.278]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -1.295% CI [-1.488, -1.104]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -1.295% CI [-1.472, -1.100]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -1.195% CI [-1.448, -1.034]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -1.195% CI [-1.194, -0.957]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -1.195% CI [-1.367, -0.913]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -1.395% CI [-1.504, -1.136]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.895% CI [-0.954, -0.628]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -1.195% CI [-1.226, -0.969]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.795% CI [-0.903, -0.654]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.895% CI [-0.917, -0.600]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.895% CI [-0.974, -0.692]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -0.295% CI [-0.450, +0.132]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +1.295% CI [+0.996, +1.528]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +2.995% CI [+2.356, +3.624]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | |
| Haiku 4.5 B -2.11 | -2.495% CI [-2.551, -2.199]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -2.695% CI [-2.840, -2.404]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -2.695% CI [-2.806, -2.384]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -2.395% CI [-2.416, -2.127]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -2.395% CI [-2.441, -2.090]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -2.495% CI [-2.562, -1.958]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -2.395% CI [-2.422, -2.190]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -2.695% CI [-2.766, -2.458]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -2.295% CI [-2.313, -2.098]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -2.195% CI [-2.285, -1.931]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -2.095% CI [-2.209, -1.891]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -1.995% CI [-2.081, -1.683]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -1.895% CI [-1.956, -1.572]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -1.595% CI [-1.666, -1.131]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -1.295% CI [-1.528, -0.996]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | +1.695% CI [+1.153, +2.337]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | |
| Haiku 4.5 A -3.46 | -4.195% CI [-4.450, -3.364]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -4.595% CI [-5.032, -3.828]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -4.095% CI [-4.633, -3.623]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -3.495% CI [-4.097, -3.292]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -3.895% CI [-4.901, -3.477]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -3.795% CI [-4.073, -3.157]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -3.295% CI [-3.442, -3.022]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -4.195% CI [-4.519, -3.545]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -3.895% CI [-4.347, -2.753]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -4.095% CI [-4.611, -3.267]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -3.395% CI [-4.308, -2.857]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -3.195% CI [-3.906, -2.922]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -3.495% CI [-3.784, -2.453]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -3.295% CI [-3.818, -2.388]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -2.995% CI [-3.624, -2.356]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs | -1.695% CI [-2.337, -1.153]20 complete seed pairs; 40 scored / 40 attempts; 0 failed, including 0 hangs |
The climb
Practice margin vs games spent
17 lines show the authoring sessions: practice margin per eval call against cumulative games spent (of 300). Diamonds show each submitted bot's final expanded-league score. Practice and league use different maps and opponents, so their scores describe different evaluations. Hover a model in the legend to isolate its runs.
How it works
The protocol, in four cards
One session, one bot
Each run is a single non-interactive agentic session in the model's native CLI
(DeepSeek official Harness at max; frozen Claude Code and Codex runs at xhigh). It gets the
rules, a spec, an eval tool and up to 300 practice games, with no wall-clock cap. Whatever
submission.py holds when the session ends enters the league.
Flash A used 296/300 games, including 7 charged, unscored 70-second engine wall timeouts. B submitted after 236/300 with no practice failures. These authoring outcomes are separate from the league results.
Pro A submitted after 300/300 practice games, with 0 practice failures and 0 hangs. Only this completed authoring run enters the Pro result; B is not included.
The game film
After every eval, per-game digests land in the workspace: one line per decision plus fight, capture and sighting events — everything the bot could legally observe, nothing more. Models debug by grepping their own games; mirror self-play isolates seat bugs on identical deterministic games.
Generated maps, sealed league
Practice runs on 12 randomly generated mirror-symmetric maps (all 8 factions, both players the same faction per map). The league plays 8 different maps from the same generator, sealed by published SHA-256 hashes and inaccessible to entrants — overfit the practice maps and the league punishes you.
The metric
Margin = log-ratio of day-capped peak army, the game's continuous score. Every pairing schedules 20 seeds × both seats. Failed attempts are never scored or replayed; paired H2H uses only complete seed pairs. Rank = mean of a model's available run medians. All N=1 rows are single-run estimates; Pro, Opus and Fable have no measured A/B gap. Eliminations are reported but not scored.
Hall of champions
Best single bot, season by season
Each season froze its own engine, sparring set and map pool; scores are only comparable within a board. Historical S9 and its title have a separate archive; earlier seasons remain in the historical snapshot JSON.
- S2 · Jul 2026 The War Season GPT-5.6-Terra on points
- S3 · Jul 2026 The Film Season Claude Opus 4.8 +0.78
- S4 · Jul 2026 The Drop-In Claude Opus 5 +0.86
- S7 · Aug 2026 Clean Slate GPT-5.6-Sol +0.86
- S8 · Sep 2026 No Clock Claude Opus 5.5 +1.20