Evaluating Generative Agents with Actions Grounded in Socially Distributed Task Environments using Incognita
06:00 · July 7, 2026 · arXiv cs.AI RSS
Effective agency in social environments depends on when an agent seeks knowledge, when it acts, and whether its actions are justified by acquired information. Existing grounded benchmarks provide executable actions, persistent state, and verifiable outcomes, while social simulation environments provide rich interaction among language agents. We study an evaluation setting that combines these requirements. We define socially distributed task environments as interactive environments where task-relevant knowledge is partitioned across role-isolated participants and consequential actions are accessible only through them. Communication serves as exploration over role-partitioned knowledge, while grounded action serves as exploitation over environment state. We introduce Incognita, a Concordia-based framework that separates social interaction from grounded execution. The evaluated agent routes messages to a user or specialist entities; specialists mediate admissible operations; a deterministic sub-environment executes accepted operations over a canonical state; and an offline evaluator scores outcomes with inherited rewards. Incognita-Retail transforms tau-bench retail into a multi-entity environment while preserving final-state reward semantics. We evaluate three generative agent models on 18 tasks stratified by social breadth, with 540 trials. Progress appears in reward and behavior: success rises from 0 percent to 8.9 percent and 17.2 percent, while premature finalization falls from 100 percent to 87 percent and 58 percent. Stronger models elicit more hidden knowledge, contact more entities, and attempt more grounded writes, yet reliability remains low. These findings show that socially distributed task environments expose behavior before reliable success, including knowledge elicitation, source selection, grounded action attempts, and premature completion belief.
Summary
Effective agency in social settings requires agents to decide when to gather information, when to act, and whether their actions are supported by what they have learned. Incognita addresses this by defining socially distributed task environments in which relevant knowledge is split across role-isolated participants and meaningful actions can be performed only through those participants. In this setup, communication functions as exploration across partitioned knowledge, while grounded actions serve as exploitation of the shared environment state.
The framework, built on Concordia, keeps social interaction separate from execution. The agent under test routes messages to a user or to specialist entities; the specialists approve only admissible operations; a deterministic sub-environment applies those operations to a canonical state; and an offline evaluator scores results using inherited reward signals. Incognita-Retail adapts the tau-bench retail domain into this multi-entity format without changing the final-state reward semantics.
Three generative agent models were tested on 18 tasks stratified by social breadth, for a total of 540 trials. Stronger models produced measurable gains: task success rose from 0 percent to 8.9 percent and then to 17.2 percent, while premature finalization dropped from 100 percent to 87 percent and 58 percent. These models also contacted more entities, elicited more hidden information, and attempted more grounded state changes. Even so, absolute reliability stayed low, indicating that the environments surface differences in knowledge-seeking and justification behavior well before they produce dependable performance.
Why it matters
This research provides a novel evaluation framework for multi-agent systems, which is highly relevant for Dutch AI researchers focusing on reliable and interactive AI. It offers actionable methodologies for testing agent behavior in complex, socially distributed environments.
