Behavioral Controllability of Agentic Models for Information Extraction: From Fixed Workflows to Reflective Agents
06:00 · July 20, 2026 · arXiv cs.AI RSS
Large language model (LLM) agents are increasingly used for complex information-extraction tasks, yet it remains unclear whether agentic components such as reflection and memory lead to observable and controllable improvements over fixed LLM workflows. We study this question through conference-paper dataset extraction, where a system must identify datasets mentioned in scholarly PDFs and produce structured records. We compare a fixed workflow baseline with reflective agent variants and specify an optimized agent condition (S2) that extends the same task with richer PDF tools and dynamic tool selection. Our evaluation emphasizes process-level behavior--including tool execution, retries, reflection, memory use, runtime, and failure recovery--while treating extraction coverage and field completeness as secondary outcome measures. The paper characterizes when agentic mechanisms change system behavior, whether these changes improve task completion, and how the observed failure modes motivate an optimized agent design under the same evaluation harness.
Summary
The paper examines whether agentic mechanisms such as reflection, memory, and retry policies produce observable and controllable changes in behavior when large language models perform structured information extraction. The concrete task is to locate dataset mentions scattered across NeurIPS 2024 PDFs and emit structured records that include name, description, task or domain, paper reference, source link, and platform. Evidence for these fields can appear in abstracts, method sections, tables, captions, references, or external URLs, often in abbreviated or ambiguous form.
A fixed workflow baseline processes each document through a predetermined sequence of prompts and validation steps. In contrast, reflective agent variants follow a ReAct-style loop that interleaves reasoning, tool calls, observation, and self-critique, optionally injecting prior extraction experiences from short- and long-term memory. An optimized S2 condition further equips the agent with twelve atomic PDF tools and dynamic tool selection, allowing it to adapt its action sequence on the basis of intermediate results rather than a static plan.
Evaluation centers on process-level traces—tool executions, reflection events, retry counts, memory accesses, runtime, and failure-recovery paths—rather than extraction accuracy alone. Coverage and field completeness serve as secondary measures. Experiments conducted with openPangu-Embedded-7B show that the richer agentic conditions alter observable behavior substantially, yet deliver only modest gains in record coverage under a fixed retry budget. The recorded traces nevertheless expose concrete failure modes that directly motivate the S2 design choices, including memory filtering and quality-aware tool policies.
The study therefore frames behavioral controllability as the degree to which an extraction system’s decisions remain inspectable, configurable through explicit parameters, and reproducible across runs. By publishing execution logs alongside the same corpus and output schema for all conditions, the work supplies a practical template for assessing when agentic components improve recovery without simply increasing runtime or trace complexity.
Why it matters
Provides rigorous, reproducible evaluation framework and design lessons for controllable LLM agents that Dutch AI researchers can directly apply or extend in information-extraction and scholarly-mining projects.
