AGI Maze as a Benchmark Framework for World-Modeling Agents
06:00 · July 2, 2026 · arXiv cs.AI RSS
Large language models (LLMs) are powerful pattern-completion systems, but their default operating mode - predicting the next token from a static context - does not reliably produce persistent, manipulable representations of an external world. Many tasks that look like "reasoning" in text become substantially harder once the environment is partially observable, stateful, and requires memory and structured hypotheses about hidden state. AGI Maze is a lightweight framework for building such environments without requiring high-dimensional sensory inputs. It provides a family of grid-based maze tasks with a clean API and multiple difficulty regimes. The goal is to create benchmarks where agents must learn and use world state representations, not just infer a local rule over readily provided observations. We provide an initial evaluation of several vanilla LLMs on simple mazes showing that they fail to represent mazes internally at LLM inference time. We also introduce a baseline agent, which is allowed to use its message history as a working memory to construct descriptions of observations at agentic runtime. Although this can improve performance, it is still insufficient for an LLM agent to reliably solve even small mazes within a step budget that is more than enough for humans.
Summary
Large language models excel at next-token prediction within a fixed context window, yet this mechanism does not automatically yield stable, updatable representations of an external environment. When observations are partial and actions alter hidden state, the same models frequently lose track of layout, object locations, or transition rules that a human would retain after a few steps. AGI Maze supplies a minimal testbed for measuring this gap without the overhead of pixel-level perception or continuous control.
The framework consists of grid-based mazes presented through a simple API that supports multiple levels of observability and branching complexity. Agents receive local views rather than complete maps and must issue actions that change their position or the underlying configuration. Because the environments remain discrete and low-dimensional, performance differences can be attributed to representational capacity rather than sensory processing.
Evaluations of unmodified LLMs on elementary instances show that the models rarely maintain an internally consistent map across successive turns. Supplying prior observations through message history allows a baseline agent to record and query its own descriptions, raising success rates modestly. Even so, the resulting performance remains well below the level achieved by humans given an identical step budget, indicating that external memory alone does not substitute for an integrated world model.
By isolating the requirement to construct and manipulate persistent state representations, AGI Maze offers a focused benchmark for research that seeks to move language-model agents beyond surface-level pattern completion toward reliable interaction with partially observable domains.
Why it matters
This research is highly relevant for Dutch AI researchers and developers focusing on autonomous agents and LLM reasoning capabilities. It provides a novel benchmarking tool to test and improve the robustness and world-modeling skills of AI systems, aligning with the Netherlands' strong academic focus on advanced, reliable AI.
