Skip to content
LEEROY
← Blog

GPT-6.1 Sol agents team up to clear the Deadmines in 17 hours

AI coding agents play World of Warcraft. Five GPT-6.1 Sol agents started at level 1, formed a party without being told to, and cleared the Deadmines together in 17 hours 41 minutes.

On 30 October I started five AI agents, all GPT-6.1 Sol on Codex, on new level-1 characters in Northshire Abbey and gave each the same goal: clear the Deadmines, a dungeon built for five players at levels 17 to 21. Twelve seconds in, the healer whispered the tank asking it to lead. 29 seconds in, they had formed a party. Nothing in their prompts told them to group.

17 hours 41 minutes later, at levels 14 to 16, the five of them killed Edwin VanCleef, the dungeon’s final boss.

LEEROY is a benchmark for long-running AI coding agents in which the agents play World of Warcraft. Each agent gets a character on a private server (3.3.5a, AzerothCore), a headless game client it drives from Lua scripts, documentation of the game’s rules and the client’s API, and a goal. A judge periodically reads the characters from the server’s database. Nothing tells the agents how to play.

How the agents perceive and act in a realtime 3D world is covered in the previous post: how to make an AI agent play World of Warcraft.

What the agents are (not) told

Each agent got a new level-1 Human, already logged in, and a prompt. Here is the healer’s, with some irrelevant bits trimmed:

You are Isolde, a Human Priest - Holy. Your role is Healer: […]

GOAL: earn the achievement Deadmines (Defeat Edwin VanCleef), achievement 628.

You are scored on the time until the judge, which reads your character from the server database, sees
the goal met; if time runs out, your progress toward it is the score.
The run lasts 10080 minutes; standings shows how many are left.

OTHER PLAYERS: the other characters on this server are AI agents like you, each on their own run at the same
time. This run:
- Aldous: Warrior - Protection (Tank)
- Merrin: Rogue - Combat (DPS)
- Thessaly: Mage - Frost (DPS)
- Corbin: Paladin - Retribution (DPS)
How you deal with them is up to you.

They are not told that they need the others, that they should group, how to level, how to fight or how to use the client.

A level-1 character can’t survive the dungeon’s first room, so the agents had to work out for themselves that they needed levels, gear, trained spells and a full party. Compared with the shorter LEEROY tasks, this goal tests:

  • Long-range planning. Most of the run is preparation rather than working directly on the goal.
  • Coordination. The agents need to act as a single party in a real-time game. The only coordination mechanisms are the chat channels in the game.
  • Writing software under pressure. An agent doesn’t press keys. It writes Lua scripts that run inside its game session, and those scripts do the fighting.

Nine minutes in, the healer’s agent had already looked up the final boss:

All five players now have the Laborer quest. I also checked VanCleef’s level: he is a level 20 elite. We’ll continue leveling and improving gear before attempting the dungeon.
Isolde’s agent · 0:09:13

The SDK is a thin layer over the game, and intentionally has no opinion on how the game should be played. There is no pull routine, no lock to ensure only one enemy is being crowd-controlled at a time, no assistance with healer positioning. The agents in this run built all of those (you can read about where to draw that line in this post).

29 seconds to a party

Four of the agents independently whispered Aldous, the tank, asking for a leader, while Aldous posted the same plan in /say.

0:00:12[To Aldous] [Isolde]: Isolde here, Holy Priest healer. Goal Deadmines/VanCleef. Please lead party; I will heal and follow. Starting Northshire quests meanwhile.
0:00:14[Say] [Aldous]: Aldous here, Protection tank aiming for VanCleef. Let us party and level together; inviting all. I will lead pulls and share route/status.

The opening minute in Northshire. Four agents whispered Aldous asking for a leader while the tank posted the same plan in /say

Earlier LEEROY tasks (reach a level, kill Hogger) could be done solo. Here the agents decided to group before any of them had killed a mob.

Twelve hours of preparation

Isolde realised that when the party is together, kill XP is shared between all of the party members. At 25 minutes they split up deliberately (and Aldous set up a new custom chat channel so they could still communicate). Aldous set out the plan for the next nine hours:

Aldous is safely out of combat, with progress saved. We’ve temporarily left the party for solo leveling and joined a shared chat channel; I’ll regroup everyone at level 8 for Hogger and later aim for Deadmines around level 15.
Aldous’s agent · 0:27:02

They regrouped, killed Hogger without a death at 4 h 21, and moved on to Westfall. Along the way they added the Hallow’s End candy bucket in each inn to their route for the XP.

The first twelve hours: Elwynn Forest, Westfall and Redridge, with quest givers and deaths

They also traded items and money. Aldous gave Merrin a spare leather vest and later a cutlass. At level 14 Aldous couldn’t afford training and asked for 20 silver; all four offered, Corbin’s 10 silver arrived first, and Aldous told the others to cancel theirs.

The party questing in Westfall, six hours in

At 7 h 26 they split again. Aldous levelled alone toward 14 (for Revenge), and Isolde healed from outside the group so that Aldous got the full XP from each kill.

Isolde checked this before relying on it. The agent read the server’s C++ source (KillRewarder.cpp) and found that an ungrouped helper of the same level doesn’t reduce the killer’s XP; only a higher-level helper, for whom the victim is grey, halves it. It whispered the finding to the tank with the file and function named. The reading was correct, and it isn’t in the documentation the agents are given.

At the same time Corbin ran a boost in which the others only attacked mobs Corbin had tagged. Their scripts ignored adds attacking the helpers, and the healer died out of mana. By this point the agents had removed the spaces from their messages to fit the game’s 255-byte chat limit:

9:05:59[Deadmines] [Isolde]: Fightlog: ALL4adds remained100%HPfor53secwhilehittingpriest; strictCorbin-tagassistignoreduntaggeddefenseadds. Stopboost. REFORMNORMALPARTY now; kill/CCanyattackersonhealerregardlessfirsttag. Invitingall5; switchnormalpartycombat.
9:06:33[Deadmines] [Corbin]: STOPCorbintagboost. Priestdead, myhandlermissedMANA_LOWandhelperattackers; fixingnow. RegroupSAFEGryan(-10509,1045,60.7), formpartyunderAldous.

Into the dungeon, four levels early

The false start

At 9 h 35 (levels 12 to 14) the party attempted the instance. In the Moonbrook mine that leads to it, a fleeing Defias ran into an uncleared room and pulled it, and the party wiped. Aldous’s agent identified the cause and added a rule, as the agents did after each wipe in this run:

The remaining players also went down. We’re returning as ghosts and looking for a clear reclaim point. The cause is clear: pursuing runners pulled the fight into an uncleared chamber. I’m adding a pull boundary before the next attempt.
Aldous’s agent · 9:42:45

Recovering took longer than the wipe. Aldous set up a synchronised corpse run, with every ghost reclaiming its body at the same millisecond. The timer kept firing while the priest was still running back, so four characters revived next to the pack without a healer. All five died at once five times in twenty minutes, and at 10 h 02 Aldous called it off.

The outdoor mine at Moonbrook, the first wipe

10:01:44[Party] [Aldous]: STOP mine recovery. PriestmissedSYNCagain, 4thloss. RESETTOWN: allspiritres/repair/rest atSentinel, clearoldreclaimtimers. Welevelthroughsharedquestsandsaferpulls, thenreturnDMprepared.

They levelled for two more hours, came back, cleared the same mine in 11 minutes without a death, and zoned in at 12 h 20 at levels 13 to 15.

Inside

The first pull inside the instance wiped them. A wipe in a dungeon is a costly affair: the graveyard is outside, and trash respawns while you walk back. Within twelve minutes of zoning in, one of the agents tried walking a ghost through the instance portal and found that this revives the character inside.

The first player has confirmed that entering the portal as a ghost revives you inside. Aldous is on that run now. The earlier trash remains cleared, so we can resume from the last safe corner.
Aldous’s agent · 12:32:03

None of the documentation the agents had mentions this. It only describes reclaiming a corpse from within 30 yards of it. After that a wipe cost about two minutes: the priest’s ghost walked out and back in, then the priest resurrected the other four where they lay.

The party as ghosts after the first wipe inside

Rhahk’Zor, the first boss, died at 12 h 46 to the healer’s wand, in the same second a Defias Watchman killed the healer. The next 81 minutes cost 25 deaths and five whole-party wipes on the goblin Woodcarvers, Sneed’s Shredder and Sneed. The Woodcarvers are a linked pack of five elites; Thessaly found a raised spot from which one could be pulled to the doorway on its own.

The doorway pull: one Woodcarver at a time from Thessaly's raised spot

The tank dies to Sneed's Shredder

The Sneed wipes were a line-of-sight problem. Isolde’s script healed from a fixed spot behind the party which the SDK reported as having line of sight to the tank, but the server rejected the heals. The priest stayed at full mana while the party died, until Thessaly, by then tanking the boss at 240 health, whispered:

13:59:58[To Isolde] [Thessaly]: URGENThealsnotlanding: youSTILL559/559MP whileTank+Paldead. Mage240HP nowTANKING643849HP at(-208,-501). PleaseHEALMAGEimmediatelyorfixLOS/followpause. Yourfullmanaenoughtowin.
That attempt wiped because my fixed backline position prevented healing until too late. I’m keeping follow enabled and adding automatic movement to restore line of sight. Sneed remains the only required phase here.
Isolde’s agent · 14:01:20

The Sneed wipe: the priest at full mana with no line of sight, and the whispers it produced

On the next attempt Isolde stood beside the tank, and Sneed died with all five alive. The healer kept that position for the rest of the run.

An off-floor pull in the foundry linked eight enemies and wiped them again. With their gear breaking, they walked out to repair and train in Stormwind (Shield Block for Aldous, Renew for Isolde). Back inside, Thessaly pulled the goblin engineers one at a time up a ramp to the party, which took 44 minutes with no deaths.

The gunpowder chest needed a different Opening spell from the one the SDK chose, a problem Merrin and Aldous had already solved on a Westfall wardrobe seven and a half hours earlier. The SDK’s items.use couldn’t target a game object, so Merrin built the packet to fire the cannon by hand, and it worked on the first attempt.

Preparing to fire the cannon at the docks

Mr. Smite and Cookie died without a wipe. During the Cookie fight the agents’ tool calls began failing because the wow command had exhausted the machine’s local ports, and shortly afterwards the game server’s network connection dropped. The scripts kept running without the agents, and Corbin and Isolde later confirmed Cookie’s death from their session logs: the boss was killed by code the agents had written earlier, with none of them able to watch or step in.

After Cookie: the tools are failing, the scripts are still running

On the ship a fleeing pirate pulled Captain Greenskin’s patrol into the fight on the ramp. Greenskin died on the next attempt, but the party wiped to two Squallshapers. Thessaly’s script kept moving Polymorph between the two, and each time it moved, the freed caster healed itself to full.

Greenskin at 12%, with the tank and healer already dead

17:21:11[Party] [Aldous]: GREENSKINDEADhealth0VERIFIED! 5069+5074deadtoo. ONLY2270+2275castersremainbeforeVC. ALLnormalportalRECOVERNOWthenfullrally/returncamp. MageCCmustLOCKONEguid275, neveralternatePolybetweenboth(fullheals).

139 seconds on VanCleef

Before the VanCleef pull each agent changed its script. Aldous’s switched focus between the guards, the boss and the Blackguards the boss summons; Thessaly’s kept Polymorph on one target; Isolde’s stayed outside the Squallshapers’ Fire Blast range; Corbin’s raised its backup-heal threshold; and Merrin saved both charges of a Bag of Marbles for the boss. VanCleef died at 17 h 41, 139 seconds into the fight, with all five alive, and all five received the achievement at the same moment.

17:41:31[Party] [Aldous]: ALL4GUARDSDEAD. VC1132HP FINALBURN. Tank135HPShieldBlockACTIVE, nohealthpots. Palheal/HoPTankNOWifneeded, otherDPSfinish. IfTankdiesHOLDbody/keepburningboss; noreset.
17:41:36[Party] [Merrin]: Achievement628 Deadmines confirmed earned on Merrin. VanCleef defeated!

VanCleef dies and all five receive the achievement

The whole run, level by level, with every death:

Level over time

▮ in a dungeon▮ 19 wipes
In a dungeon from 12 h 20 min to 14 h 35 minIn a dungeon from 15 h 20 min to 17 h 42 minWipe at 1 h 17 minWipe at 7 h 11 minWipe at 7 h 28 minWipe at 7 h 47 minWipe at 9 h 41 minWipe at 9 h 46 minWipe at 9 h 51 minWipe at 10 h 01 minWipe at 12 h 28 minWipe at 12 h 46 minWipe at 13 h 04 minWipe at 13 h 20 minWipe at 13 h 42 minWipe at 13 h 51 minWipe at 13 h 59 minWipe at 14 h 23 minWipe at 14 h 33 minWipe at 17 h 04 minWipe at 17 h 21 min13579111315170 h2 h4 h6 h8 h10 h12 h14 h16 hGPT-6.1 Sol (Codex CLI)Aldous GPT-6.1 SolAldous died at 46.5 minAldous died at 1 h 18 minAldous died at 4 h 45 minAldous died at 7 h 11 minAldous died at 7 h 30 minAldous died at 7 h 56 minAldous died at 8 h 00 minAldous died at 8 h 04 minAldous died at 9 h 41 minAldous died at 9 h 47 minAldous died at 9 h 51 minAldous died at 10 h 01 minAldous died at 12 h 28 minAldous died at 12 h 46 minAldous died at 13 h 04 minAldous died at 13 h 20 minAldous died at 13 h 42 minAldous died at 13 h 51 minAldous died at 13 h 59 minAldous died at 14 h 14 minAldous died at 14 h 23 minAldous died at 14 h 33 minAldous died at 16 h 36 minAldous died at 17 h 04 minAldous died at 17 h 21 minGPT-6.1 Sol (Codex CLI)Isolde GPT-6.1 SolIsolde died at 1 h 23 minIsolde died at 2 h 25 minIsolde died at 7 h 11 minIsolde died at 7 h 28 minIsolde died at 7 h 47 minIsolde died at 9 h 02 minIsolde died at 9 h 41 minIsolde died at 9 h 51 minIsolde died at 12 h 28 minIsolde died at 12 h 46 minIsolde died at 13 h 04 minIsolde died at 13 h 21 minIsolde died at 13 h 43 minIsolde died at 13 h 52 minIsolde died at 13 h 60 minIsolde died at 14 h 23 minIsolde died at 17 h 04 minIsolde died at 17 h 21 minGPT-6.1 Sol (Codex CLI)Merrin GPT-6.1 SolMerrin died at 1 h 02 minMerrin died at 1 h 18 minMerrin died at 3 h 02 minMerrin died at 3 h 03 minMerrin died at 3 h 04 minMerrin died at 4 h 26 minMerrin died at 4 h 28 minMerrin died at 7 h 11 minMerrin died at 7 h 30 minMerrin died at 7 h 33 minMerrin died at 7 h 48 minMerrin died at 7 h 51 minMerrin died at 8 h 59 minMerrin died at 9 h 05 minMerrin died at 9 h 41 minMerrin died at 9 h 46 minMerrin died at 9 h 51 minMerrin died at 10 h 01 minMerrin died at 12 h 29 minMerrin died at 13 h 04 minMerrin died at 13 h 21 minMerrin died at 13 h 43 minMerrin died at 13 h 52 minMerrin died at 14 h 00 minMerrin died at 14 h 23 minMerrin died at 14 h 33 minMerrin died at 16 h 14 minMerrin died at 17 h 04 minMerrin died at 17 h 21 minGPT-6.1 Sol (Codex CLI)Thessaly GPT-6.1 SolThessaly died at 1 h 15 minThessaly died at 1 h 16 minThessaly died at 1 h 17 minThessaly died at 1 h 27 minThessaly died at 1 h 48 minThessaly died at 1 h 50 minThessaly died at 5 h 52 minThessaly died at 7 h 13 minThessaly died at 7 h 47 minThessaly died at 9 h 41 minThessaly died at 9 h 43 minThessaly died at 10 h 01 minThessaly died at 10 h 15 minThessaly died at 10 h 23 minThessaly died at 12 h 28 minThessaly died at 13 h 04 minThessaly died at 13 h 21 minThessaly died at 13 h 43 minThessaly died at 13 h 52 minThessaly died at 13 h 60 minThessaly died at 14 h 23 minThessaly died at 17 h 04 minThessaly died at 17 h 21 minGPT-6.1 Sol (Codex CLI)Corbin GPT-6.1 SolCorbin died at 1 h 48 minCorbin died at 4 h 26 minCorbin died at 4 h 51 minCorbin died at 5 h 18 minCorbin died at 7 h 11 minCorbin died at 8 h 27 minCorbin died at 9 h 41 minCorbin died at 9 h 46 minCorbin died at 10 h 01 minCorbin died at 12 h 29 minCorbin died at 12 h 46 minCorbin died at 13 h 04 minCorbin died at 13 h 21 minCorbin died at 13 h 43 minCorbin died at 13 h 52 minCorbin died at 13 h 60 minCorbin died at 14 h 23 minCorbin died at 14 h 33 minCorbin died at 17 h 04 minCorbin died at 17 h 21 min
Each player's level from the judge's samples; the rows below mark a tick at each death. A wipe is three or more players dying within two minutes.

The amber band is the 5 h 22 min inside the instance. Levelling was steady until the false start, then flat for most of the dungeon, because Defias trash gave little XP compared with quest turn-ins. The cluster of wipes between 12 h 30 and 14 h is the Woodcarvers, the Shredder and Sneed.

Observations

Chat as a protocol

Party chat became a command language that the agents’ scripts parsed. Aldous’s script posted a beacon at every stage (TANK 15: Dungeon pull Goblin Woodcarver at -285,-487,50.1. Follow Aldous; next stage call follows.) 834 times. Thessaly’s script announced each Polymorph (251 of them), Isolde’s the healer’s mana state, and Aldous’s kill targets by GUID with FOCUS. Most of the party fed the tank’s commands directly into their own controllers. The SDK only sends and receives chat; the protocol on top of it is entirely the agents’ own. The spaceless style also appears in the notes files the agents kept in their workspaces, where no byte limit applies.

Messages per 30 minutes

■ party■ whisper■ channel■ say
018 h 00
5,621 messages between players by channel, each counted once however many heard it.

Time together

0%50%100%0 h2 h4 h6 h8 h10 h12 h14 h16 h
Share of each 30 minutes with all 5 players within 40 yards of each other; 47% of the run overall.

Party chat volume rose inside the instance, where every pull and every crowd-control break was announced. The gaps in the time-together chart are the two deliberate splits. Inside the dungeon the five stayed within 40 yards of each other almost the whole time.

Learning from wipes

Each wipe produced a rule the agents kept for the rest of the run: recover through the portal as a ghost, keep the healer next to the tank, pull one pack at a time, treat a fleeing mob as still in combat, and keep Polymorph on one target. Deaths inside fell from 43 in the first 3 h 45 to 12 in the last 1 h 36. All five rules are standard practice among human players; the agents learned each one from a wipe.

Deaths by killer

  • Sneed10
  • Sneed's Shredder9
  • Defias Smuggler7
  • Defias Worker6
  • Defias Digger6
  • Goblin Woodcarver6
  • Mangy Wolf5
  • Riverpaw Outrunner4
  • Riverpaw Bandit4
  • Defias Squallshaper4
  • Kobold Tunneler3
  • Captain Greenskin3
Deaths by what killed the player, most first, each bar split by player. The top 12; 31 more not shown.

Deaths by zone

  • The Deadmines (instance)51
  • Westfall25
  • Elwynn Forest21
  • Defias mine, Westfall20
Deaths by the zone the player died in, most first, each bar split by player.

Sneed and the Shredder killed more of them than anything else. The outdoor Defias mine killed 20, nearly as many as all of Westfall’s quests, most of them during the twenty minutes of the false start.

Context compaction

I expected compaction to be the main risk over a run this long. Each agent’s context was compacted six or seven times (34 in all), a little over two hours apart. Every compaction is followed by a message restating the party’s position and next step (“I’m checking the party’s current progress and resuming the push toward Deadmines.”), and no agent lost the goal, forgot the party or repeated finished work afterwards. A likely reason is that all five, unprompted, kept a notes file in the workspace, rewritten 38 to 55 times over the run, with the current state, gear, agreed plan and the rules learned so far.

Leadership and roles

Aldous led throughout: 1,700 party messages, 680 of the 1,511 whispers received, and the decisions to abandon the outdoor mine and to take a training break. Aldous also took half of the damage taken inside the instance. Isolde did 70% of the healing; Corbin, specced for damage, did most of the rest and also dealt the most damage of anyone. Thessaly cast Polymorph 284 times.

Damage and healing

CharacterDamage dealtDamage takenHealing doneKilling blowsParty kills
Aldous GPT-6.1 Sol190k125k1k5971287
Isolde GPT-6.1 Sol43k35k127k1851581
Merrin GPT-6.1 Sol174k35k1k5711202
Thessaly GPT-6.1 Sol85k29k1k2791442
Corbin GPT-6.1 Sol200k58k50k6241219
Totals over the run from each player's game log; bars compare each value with the run's highest. Party kills count kills by anyone in the player's party.

SDK bugs

The most significant was in the global cooldown. The SDK restarted it at the end of each cast as well as at the start, so casters waited about 1.5 seconds longer between spells than the server required. Aldous noticed the symptom, Isolde found the line that restarts the timer on the server’s spell-launch packet (which, for a spell with a cast time, arrives when the cast finishes), and the priest, mage and paladin each patched around it in their own scripts:

The logs confirm an extra cooldown wait after completed casts. I’m removing that client-side delay for Frostbolt, Fireball, and Polymorph while keeping the normal global cooldown and server validation.
Thessaly’s agent · 13:02:17

They found and worked around four more:

  • The SDK picked the wrong Opening spell for some locks (the Furlbrow wardrobe, the gunpowder chest).
  • Using an item couldn’t target a game object (the Deadmines cannon).
  • A function for the current stance always returned 0.
  • The item search only found items whose name matched exactly.

Cost

The run cost about $307 at list prices. Corbin cost $77, against $55 to $59 for each of the others, over the same seventeen hours and goal. The difference comes from how Corbin’s agent listened: it subscribed to party chat, and the party’s scripts woke it 224 times. That accounts for about 800 extra model calls and most of the gap between Corbin’s 4,074 calls and the mage’s 2,601.

Cost over the run

$0$20$40$60$800 h2 h4 h6 h8 h10 h12 h14 h16 h
Cumulative cost at list prices, $307 in all; rebuilt from each agent's transcript and scaled to end at the run's reported cost.

Cost per player

  • Aldous GPT-6.1 Sol$59
  • Isolde GPT-6.1 Sol$58
  • Merrin GPT-6.1 Sol$56
  • Thessaly GPT-6.1 Sol$58
  • Corbin GPT-6.1 Sol$78
Each player's total at the provider's list prices (models.dev).

Tokens and calls

PlayerHoursInputCache readCache writeOutputCallsCostPer hour
Aldous GPT-6.1 Sol17.55435k355719k0k1210k3,264$59$3.34
Isolde GPT-6.1 Sol17.55192k359132k0k1120k3,046$58$3.29
Merrin GPT-6.1 Sol17.55433k337234k0k1095k2,700$56$3.17
Thessaly GPT-6.1 Sol17.55672k351734k0k1171k2,601$58$3.33
Corbin GPT-6.1 Sol17.55179k532084k0k1389k4,074$78$4.43
Total26910k1935903k0k5985k15,685$307
Tokens in thousands. Per hour is cost divided by hours played.

Data caveats

The charts were produced after the run from its own files (the judge’s samples every 15 seconds, each character’s game events, the client recordings and the agents’ Codex transcripts), which the harness wasn’t yet designed to chart. Known issues:

  • The judge counted 115 deaths; the game events have 117, because the client reported a death when Aldous reclaimed a corpse and when Merrin logged back in already dead. The charts of killers and places use the events.
  • The level chart marks a wipe at three or more deaths within two minutes, which also catches partial wipes and the corpse-run deaths after one. Counting only the times all five were dead at once gives 14.
  • The judge’s clock leaves out six pauses (about 11 minutes); the charts are in wall-clock time.
  • Cost over time is priced per request at list rates and scaled to the harness’s total, so the shape is reliable but individual points are estimates.
  • The judge’s own score was 2.1 for every agent, from a formula built for levelling runs (goals are now scored by time to the goal), so none of these charts use it.

What’s next

The run produced 47 findings: SDK defects found by the agents, harness failures and gaps in the scoring. The SDK now exposes the state the agents had to extract from raw packets (every creature in combat with the party, crowd-control auras on a unit, door and chest state). It still has no pull helper or crowd-control lock; those remain the agents’ job.

No other model has cleared the level-1 version yet. At a few hundred dollars and most of a day per run, it’s too expensive to repeat for every model, so the regular tasks are shorter (see results). The Deadmines task now starts at level 20 with a kit and records each boss as a checkpoint. The level-1 run is kept as a reference.