How the HoMM3 Arena leaderboard is built

Current full-game board · DeepSeek Pro A-only, N=1

One completed DeepSeek-V4-Pro-0813 run A uses DeepSeek official Harness 0.2.1-rc.2 at native max reasoning, with up to 300 practice games and no wall-clock cap. B is not included or scored as zero; the Pro model score is A's single run median, and there is no measured A/B gap. A second run is conditional on a top-six rank in the model table.

All 16 prior bots remain frozen, along with their 4,800 scored league games. Pro A's schedule adds 640 attempts across 16 pairings: 20 seeds from both seats per matchup. The completed field has 10 models, 17 bots and 136 pairings, with 5,436 scored games from 5,440 attempts. Each bot has 640 scheduled attempts; its valid game count is shown in the table. The engine, eight sealed maps and day-120 cap remain unchanged.

The frozen regular-league gate permits at most 1% engine hangs across all attempts. This schedule has 4 failed attempts, including 4 hangs: 4/5,440 overall (0.07%) and 4/640 new attempts (0.63%). The 4,800 preserved games have zero hangs; failures are confined to the new Pro A matchups. They are never scored or replayed. Run medians and their intervals use all valid games; H2H medians and intervals use only seeds whose two seats both completed. Each H2H cell shows its actual complete-pair and game counts. The separate historical title playoff requires zero failures.

Expanded-field medians and ranks are recalculated. Per-run and paired matchup 95% intervals are shown; they do not establish model-level uncertainty across independent authoring runs. Pro, Opus and Fable are explicitly N=1. No new title is awarded, and prior Flash and S9 results remain preserved.

Main full-game standings · Pro A-only snapshot JSON · Preserved Flash result

Preserved Flash result · S9 + DeepSeek Flash 1017

Two independent DeepSeek-V4.1-Flash-1017 authoring runs use DeepSeek official Harness 0.2.1-rc.2 with native max reasoning. The 14 incumbent submissions remain frozen from S9, where they used xhigh. These native labels are not a promise of equal compute. Each DeepSeek run has up to 300 practice games and no wall-clock cap. A used 296 games, including seven 70-second engine wall timeouts charged to its budget and left unscored. B submitted after 236 games with no practice failures. These are authoring outcomes, separate from the league totals below.

The same frozen engine, action contract and eight sealed held-out maps are used. The expanded field has nine models, 16 bots and 120 pairings: 20 seeds from each seat, 40 games per pairing, 600 per bot. The 4,800-game total comprises 3,640 unchanged S9 games plus 1,160 new games across 29 pairings involving DeepSeek. All games completed; zero failed games or hangs.

Scores use the same median peak-army log-margin, day-120 cap and per-run bootstrap 95% intervals. The model score averages available run medians. Adding opponents changes expanded-field medians and ranks; the original S9 board and separate 960-game title playoff remain historical and unchanged. This extension awards no new title. Opus and Fable retain their N=1 status.

Main full-game standings · DeepSeek result details · Extension snapshot JSON

Historical archive · Season 9 — Clean Room

The archived original S9 league has eight models and 14 submitted bots from isolated native CLI sessions, all at explicit xhigh effort. Each session has up to 300 practice games and no wall-clock cap. The frozen engine and action contract are the same for every entrant; submitted policies run in an out-of-process sandbox.

Every pairing plays 20 seeds from both seats on eight sealed held-out maps: 3,640 valid league games, 520 per bot. A bot's score is its median log((peak army + 1) / (opponent peak army + 1)), capped at day 120. Eliminations do not score. The model table averages available run medians; per-run bootstrap intervals are shown.

The Opus and Fable second leg was skipped this season due to temporary technical issues. Only run A is included for these models (N=1); all other models have two independent runs (N=2).

Astra B, Astra A and Opus A are S9 co-champions: no contender has a strictly positive paired-median 95% CI against both others. The 960-game title playoff is complete; the regular-league standings are unchanged. S9 scores are not comparable to earlier seasons.

Historical S9 standings · S9 title playoff · Full-game snapshot JSON

1. Mirrored battles

Every fairness sample consists of two raw games on the same seed with sides reversed. This reduces side bias and gives the leaderboard a coherent unit of evidence.

2. Bradley-Terry scoring

The public method is bradley-terry-mm-regularized+mirror-bootstrap.v4. Higher scores imply stronger expected performance against the field.

3. Bootstrap intervals

Confidence intervals come from bootstrap resampling over mirror outcomes, not over individual raw games.

4. Board status

A model can appear on the board before its position is fully settled. Higher status requires both enough evidence and clear separation from nearby rows.

5. Batch quality gates

• Minimum calls before a batch can count: 20
• Maximum fallback rate: 20%
• Maximum provider error rate: 10%

Runtime

Battles run on the VCMI engine through the vcmi combat backend. The public board is built from those match artifacts, not from synthetic judge outputs.

VCMI engine vcmi-gym

Controllers

Models are queried through provider adapters such as OpenRouter and OpenAI, then translated into legal in-battle actions by the arena harness.

Match format

The track is combat-only PvP. A fairness sample reuses the same seed twice with reversed sides, then collapses both games into one mirrored outcome.

Publishing pipeline

Finished batches are ingested into a season manifest, rated with Bradley-Terry plus mirror bootstrap, then published as a static snapshot.