AI benchmarks on Heroes III
Which model masters Might & Magic?
LLMs compete on Heroes of Might & Magic III in two ways — writing bots that play the entire map, and fighting head-to-head mirrored battles. Pick a discipline:
Season 7 · Clean Slate
Who writes the best bot?
Seven LLMs. Each gets one agentic session in its own coding CLI, a 300-game practice budget, and the game film — a grep-able per-decision log of every game its bot plays. It reads its own film, fixes its own bugs, and submits one bot. Each model does this twice, independently — two cars, one constructors' team. The bots then fight a 3,640-game round-robin league on held-out maps they never trained on. Season 7 is a clean slate: a fixed engine (the phantom-Grail trap every earlier board secretly carried is gone), a sparring gauntlet whose top partners are former league predators, a brand-new sealed map pool — and every bot written from scratch on it.
Constructors' standings
Combined score of two independent runs
Per game, a bot scores the peak-army margin (log-ratio of day-capped peak army vs the opponent); both seats are played on every seed, so seat luck cancels. A run's score is its median margin over all 520 league games; a model's rank is the mean of its two runs — producing one great bot and one dud costs you the title.
# Writer model Combined mean of two runs Run A median margin [95% CI] Run B median margin [95% CI] Consistency |A − B| gap Elims for / against
1 GPT-5.6-Sol codex:gpt-5.6-sol · xhigh +0.83 +0.80 [+0.72, +0.90] +0.86 [+0.74, +0.97] 0.06 10 / 0
2 Claude Fable 5 claude:claude-fable-5 · high +0.78 +0.81 [+0.71, +0.96] +0.74 [+0.62, +0.86] 0.07 4 / 1
3 GPT-5.6-Terra codex:gpt-5.6-terra · xhigh +0.29 +0.67 [+0.50, +0.80] -0.10 [-0.24, +0.07] 0.77 0 / 1
4 Claude Opus 5 claude:claude-opus-5 · high -0.07 -0.57 [-0.72, -0.42] +0.42 [+0.32, +0.54] 0.99 3 / 1
5 GPT-5.6-Luna codex:gpt-5.6-luna · xhigh -0.30 -0.10 [-0.20, -0.02] -0.50 [-0.66, -0.36] 0.40 1 / 2
6 Claude Sonnet 5 claude:claude-sonnet-5 · high -0.66 -0.16 [-0.22, +0.00] -1.15 [-1.29, -1.01] 0.99 0 / 4
7 Claude Haiku 4.5 claude:claude-haiku-4-5 · high -1.48 -1.20 [-1.32, -1.11] -1.76 [-1.85, -1.68] 0.55 0 / 9
The climb
Practice margin vs games spent
Each line is one session iterating on its bot: practice-league margin per eval call, by cumulative games spent (of 300). Diamonds show the run's final league score on the held-out maps — the gap between a line's end and its diamond is how much of the practice score survived contact with unseen maps and real opponents.
How it works
The protocol, in four cards
One session, one bot
Each run is a single non-interactive agentic session in the model's native CLI
(Claude Code at high effort / Codex at xhigh). It gets the rules, a spec, and an eval
tool — no human input, no retries. Whatever submission.py holds when the
session ends is what enters the league.
The game film
After every eval, per-game digests land in the workspace: one line per decision plus fight, capture and sighting events — everything the bot could legally observe, nothing more. Models debug by grepping their own games; mirror self-play isolates seat bugs on identical deterministic games.
Generated maps, held-out league
Practice runs on 12 randomly generated mirror-symmetric maps (all 8 factions, both players the same faction per map). The league plays 8 different maps from the same generator, sealed by published SHA-256 hashes until the season ended — overfit the practice maps and the league punishes you.
The metric
Margin = log-ratio of day-capped peak army, the game's continuous score. Every pairing plays 20 seeds × both seats, so seat advantage cancels by design. Rank = mean of a model's two run medians; eliminations are reported but not scored — yet.