From Black Box to Executable Logic: Explainable Reinforcement Learning through Prolog Expert Systems
06:00 · July 20, 2026 · arXiv cs.AI RSS
A trained deep reinforcement learning policy is a black box, and we ask whether it can be made explainable by rewriting it as an executable logic program that reproduces its behaviour and that a person can read, a logic engine can run, and an optimizer can edit. We present a three-stage post-hoc transformation that extracts a frozen proximal policy optimization teacher, induces an ordered rule list from its decisions in the manner of classical relational learning, and emits the result as a Prolog program whose every decision is executed by an off-the-shelf logic engine; a subsequent expansion stage edits the rule base and accepts an edit only when policy evaluation certifies a return increase. We prove four guarantees. A return-loss bound makes the distilled program a machine-checkable certificate in a finite Markov decision process, and the expansion loop improves monotonically and terminates. For the continuous-observation setting we answer whether the conversion is possible at all: the propositional threshold instantiation converts the network to arbitrary fidelity as the resolution B grows, with disagreement O(1/B) and a return gap that closes at the same rate, and a matching lower bound shows the cost is exponential in the observation dimension for an oblique decision boundary. Empirically, on a two-room key-and-door task with 16,944 reachable states the expanded Prolog program attains exact optimal return in every seed and, in a budget-capped regime, exceeds the stochastic teacher on exact return in ten of ten seeds. On three continuous-control tasks the emitted program substitutes the network, matching the neural teacher within noise on Acrobot with eleven clauses and recovering about 97% of its return on CartPole, while on the finer-control LunarLander it recovers only partially, exactly the ceiling the exponential lower bound predicts.
Summary
A trained deep reinforcement learning policy remains a numerical black box whose internal decisions resist direct inspection. The work addresses this limitation by recovering an executable first-order logic program that reproduces the policy’s behaviour while remaining readable by humans and runnable by a standard Prolog engine. The method is strictly post-hoc: a frozen proximal-policy-optimization teacher is left untouched, and the conversion proceeds in three sequential stages. First, the teacher is queried exhaustively over reachable states to obtain a census of state–action pairs. Second, an ordered list of first-order clauses is induced from these pairs by a greedy relational-learning procedure modelled on the classical FOIL algorithm. Third, the resulting rule base is subjected to a return-driven expansion loop that proposes local edits and retains only those that a policy-evaluation oracle certifies will increase expected return.
Four theoretical guarantees accompany the pipeline. A return-loss bound shows that the difference in value between the teacher and the distilled program is at most a quantity linear in the teacher’s weighted disagreement rate; in finite Markov decision processes both sides of the inequality are computable exactly by solving linear systems, yielding a machine-checkable certificate. The expansion stage is proved to improve return monotonically and to terminate. For continuous observation spaces a propositional threshold encoding is introduced that converts the network to arbitrary fidelity as resolution grows, with both disagreement and return gap shrinking at rate O(1/B). A matching lower bound establishes that any axis-aligned rule list approximating an oblique decision boundary must contain Ω(B^{d−1}) clauses, quantifying the exponential cost of the transformation in observation dimension.
Empirical evaluation is performed on a fully enumerable two-room key-and-door task containing 16 944 reachable states and on three standard continuous-control benchmarks. In the discrete environment the expanded Prolog program attains exact optimal return in every random seed; under a budget-capped teacher it surpasses the stochastic network on exact return in all ten seeds. On Acrobot the emitted program matches the neural teacher within sampling noise using only eleven clauses; on CartPole it recovers approximately 97 % of the teacher’s return. Performance on LunarLander remains partial, consistent with the exponential lower bound once finer control is required. Throughout, results are reported via interquartile means with stratified bootstrap intervals, and the authors explicitly note the vacuity of the return-loss certificate at the discount factors employed.
Why it matters
This research is highly relevant for Dutch AI researchers and practitioners focusing on Explainable AI (XAI) and compliance with the EU AI Act. By converting opaque RL models into readable Prolog rules, it offers a mathematically rigorous pathway to transparent, auditable, and safe AI systems.
