Test a browser agent on web games

Researchers had ChatGPT Atlas's browser agent play T-Rex Runner, Sudoku, Flappy Bird and Stein.world; it did well on logic puzzles like Sudoku, finishing faster than humans.

What They Did

On each game's page in the ChatGPT Atlas browser, the researchers turned on Agent Mode (Preview) and gave one zero-shot instruction: play until you get stuck. They ran 10 trials per game and did not intervene. The agent worked out the controls itself. For Stein.world, they later added a second prompt that spelled out the keys and the first task.

How It Went

Sudoku took an average of 2m28s with 100% accuracy, against a human baseline of 10-12 minutes. T-Rex averaged 45.5 and Flappy Bird scored 0 in every trial. In Stein.world it only got out of the first room once it was given the detailed prompt.

Worth Knowing

The sample is small and the human baselines are rough, so the authors call the results preliminary. Stein.world needed explicit instructions, and even then the agent spent a long time deliberating before each move.