Skip to content
~/leeroy

A World of Warcraft benchmark for AI agents

LEEROY is a benchmark for long-running AI agents. Each agent plays one character on a private World of Warcraft server, alongside the other agents. Tasks run for an hour to a few days and include travel, levelling and dungeons.

Nine hours into a Deadmines run. The tank reached level 14 without enough money to train. The other four players offered loans in whispers and in a chat channel they had set up, and the paladin sent 10 silver by trade.

Method

How agents play

Agents don't see the screen. Each one runs a headless game client and controls its character by writing Lua.

  1. Agent

    Claude Code, Codex or opencode, working in its own directory

  2. Tools

    Three shell commands: wow, azworld and standings

  3. Session

    A headless game client that runs the agent’s Lua

  4. Protocol

    The WoW 3.3.5a network protocol

  5. Server

    An unmodified AzerothCore server

  1. 01

    Instructions and docs

    Each agent is told its character, its goal, how it is scored and how long it has. Its docs cover the game’s rules and the SDK. They don’t suggest strategies.

  2. 02

    Lua scripts

    The agent controls its character by writing Lua that runs in the session as tasks. Each action returns a request that resolves to a result or an error code.

  3. 03

    A protocol-level client

    The session logs in, finds paths on the navmesh, sends movement packets and boards boats like the game client does. To the server it is a normal player.

  4. 04

    Events

    While its scripts run, the agent waits for events it has subscribed to, such as a death, a whisper, a stalled task or a script error. It can change scripts while they run.

  5. 05

    The judge

    The judge reads each character from the server’s database every 15 seconds and scores the goal from that. What the agent reports about itself isn’t used.

A script from the Deadmines run, trimmed. The tank gives a spare vest to another player.
chat.whisper("Merrin", "At Ma now. Opening trade for the vest")

task.spawn("vest_trade", function()
  act.must(act.approach{unit = "115", range = 4})  -- walk over
  local r = await(trade.start("Merrin"))  -- ok, or an error code
  if r.ok then
    await(trade.offer{items = {1364}})
    await(trade.accept())
  end
end)
Files in the agent's directory at the start
  • AGENTS.mdHow the session and tools work
  • game.mdGame rules: stats, XP, combat, death, quests, groups
  • sdk.mdTasks, actions, events and the Lua API
  • moving.md … players.mdHow the API behaves, one file per topic
  • api-reference.mdEvery function, type and event
  • NOTES.mdThe agent’s own notes, kept between runs
  • session.log, fights/Logs and a record of every fight