Agentic Knowledge Tracing: A Multi-Agent LLM Architecture for Stealth Assessment of Financial Literacy in Serious Games
06:00 · June 25, 2026 · arXiv cs.AI RSS
Assessing financial literacy during gameplay without disrupting the learning experience remains a key challenge in serious games for education. We present the Agentic BKT pipeline, a multi-agent large language model architecture for stealth assessment of financial competencies from open-ended gameplay events. The pipeline processes events from a 2D platformer serious game aligned with the OECD/INFE financial literacy framework through four phases: (1) the game captures every player decision as a structured event log; (2) an LLM event classifier labels each action on a four-point rubric validated against three domain experts (Fleiss kappa = 0.624, substantial agreement); (3) four domain-specific agents specializing in risk mitigation, investing, spending, and credit management perform session-level reasoning over behavioral trajectories, feeding per-competency Bayesian Knowledge Tracing that estimates mastery within each domain; and (4) an expert judge agent synthesizes the domain-level estimates into an overall mastery score. Evaluated with 193 K-12 participants across 264 game sessions, the Agentic BKT pipeline yields mastery estimates significantly correlated with learning gain (r = 0.276, p = 0.0001) and post-test scores (r = 0.333, p < 0.0001) while showing no correlation with pre-test scores, providing both convergent and discriminant validity. The multi-agent approach approximately triples the predictive validity of a single-LLM baseline (r = 0.095, not significant) in this study, demonstrating that domain decomposition and session-level reasoning play a central role in capturing the multidimensional nature of financial literacy from gameplay
Summary
The Agentic BKT pipeline addresses the difficulty of assessing financial literacy during gameplay without interrupting the experience. It combines a multi-agent large language model architecture with Bayesian Knowledge Tracing to derive competency estimates from unstructured player actions in a 2D platformer serious game aligned with the OECD/INFE framework. The game logs every decision—such as purchases, investments, gambling, and credit use—together with full game-state context, producing structured event sequences across sessions lasting ten to twenty minutes.
Processing occurs in four stages. An LLM-based classifier first assigns each logged action to one of four rubric levels (POOR to EXCELLENT), achieving substantial agreement with three domain experts (Fleiss’ kappa = 0.624). Four specialized agents then perform session-level reasoning over behavioral trajectories in the domains of risk mitigation, investing, spending, and credit management; each agent supplies observations to a separate Bayesian Knowledge Tracing model that tracks mastery probability within its domain. Finally, an expert-judge agent aggregates the four domain-level estimates into a single overall mastery score.
Evaluation with 193 K-12 students across 264 gameplay sessions showed that the resulting mastery estimates correlated significantly with learning gains (r = 0.276) and post-test scores (r = 0.333) while remaining uncorrelated with pre-test scores, indicating both convergent and discriminant validity. The same estimates approximately tripled the predictive strength of a single-LLM baseline (r = 0.095, nonsignificant), underscoring the value of domain decomposition and temporal reasoning for modeling the multidimensional nature of financial literacy from open-ended play.
Why it matters
This research is highly relevant for AI researchers and EdTech developers in the Netherlands, offering a novel multi-agent LLM approach to educational assessment. Its use of the internationally recognized OECD/INFE framework ensures applicability within European educational standards, providing actionable insights for deploying transparent, AI-driven evaluation tools.
