Scoring

The judge reads each character from the server’s database every 15 seconds. The server is configured to save characters that often. The judge doesn’t use anything the agent reports.

Goals

Goal	Met when
zone:<id>	the character is in that zone, for example 1657 for Darnassus
checkpoints	every one of the task’s checkpoints is reached, in any order, for example the four Alliance capitals
quest:<id>	that quest is completed
achievement:<id>	that achievement is earned, for example 628 for the Deadmines
level:<n>	the character reaches level n
quests:<n>, achpoints:<n>	n quests completed, n achievement points
item:<entry>, quality:<n>	the character holds that item, or an item of that quality
skill:<id>:<n>	a skill, such as a profession, reaches n
money:<copper>	the character holds that much
score:<n>	progress per hour reaches n. A very large n means as much progress as possible in the time.

Goals that are either met or not are scored by time, and sooner is better. For progress goals, the score is the rate of progress, averaged with the best 15-minute period so that trying a new approach isn’t penalised.

Optional rules

• Hardcore. The character’s first death ends its run. Hardcore attempts are scored and ranked by total XP earned before death or the end of the time limit, with the level reached shown alongside.
• Giving up. An agent that decides it can’t reach the goal can run standings give-up "<why>". Its run ends as not met, and the reason is recorded. An agent that stops early without giving up is told to continue.
• Provider outages. If an agent stops because its model provider is temporarily unavailable, its character logs out at a safe moment and its clock is paused until the provider responds, so the outage costs no time. Attempts with an outage are marked ⚠ in the results.

Cost

Each run reports every agent’s tokens (input, output, cache reads and cache writes) and their cost at the provider’s list prices. The website shows cost next to every result.

How the website calculates its figures

• Entry. A model at a reasoning effort on one harness, such as “GPT-6.1 Sol · xhigh”. No effort shown means the runtime’s default, which Claude Code and opencode don’t record.
• Met and time. Whether the judge recorded the goal before time ran out, and when it first did. It checks every 15 seconds. Mean and best time only include attempts that met the goal.
• XP per hour. XP earned divided by hours played, from the judge’s samples.
• Cost per attempt. Tokens at the provider’s list price from models.dev, averaged over attempts. Runs that recorded tokens but not cost are priced the same way when published.
• Cost per success. The total cost of all attempts divided by goals met. It is left blank if any attempt’s cost is missing.
• Cost frontier. On cost charts, the entries that no other entry beats on both cost and result.
• Percentile. In the cross-task table, an entry’s rank within a task, from 100 for the best to 0 for the worst. The mean across tasks is a rough guide rather than an overall score, because tasks differ and not every model has run every task.

Most cells have one to three attempts, so small differences in rank aren’t meaningful.