RENDER: Controlling Reader-Facing Evidence in LLM Memory Evaluation
06:00 · August 26, 2026 · arXiv cs.AI RSS
Memory and RAG evaluations often treat the answering model's input as an implementation detail, even though systems may render the same history as a memory entry, summary, typed record, or raw excerpt. We introduce RENDER, a benchmark control that fixes the conversation while varying the reader-facing artifact. RENDER combines a five-level packet ladder, localizing when answer-bearing content enters the input, with deterministic templates approximating ChatGPT-style entries, LangChain summaries, MemGPT-style typed records, and raw conversation. On 500 LongMemEval questions and nine models, matched-budget resolved packets beat recency-truncated raw dialogue by 42.4-72.6 points. In deployed-style templates, best-worst spread is 24.6-48.8 points per model; under the primary scorer, ChatGPT-style entries have higher point estimates than raw conversation on 7 of 9 models. Judge rescoring preserves the positive aggregate effect, but model-specific significance is mixed. Three models scoring 0 percent on formal ledger packets answer the same facts from natural-language entries at 45.4-53.4 percent. The effect persists under retrieval noise and transfers to HotpotQA, suggesting that memory/RAG evaluations should report or control the reader-facing artifact.
Summary
Memory and RAG evaluations commonly treat the input presented to the answering model as a fixed implementation detail. In practice, the same underlying conversation history can reach the model as a natural-language memory entry, a compressed summary, a structured typed record, or a raw excerpt. RENDER isolates this variable by holding the conversation, question, and answer contract constant while systematically varying only the reader-facing artifact.
The benchmark uses two instruments. A five-level packet ladder progressively introduces answer-bearing content: early levels expose only witness addresses, the middle level first writes the resolved current-state value, and later levels add metadata. Deterministic templates then render the same packets as approximations of deployed surfaces, including ChatGPT-style entries, LangChain summaries, MemGPT-style typed records, and raw dialogue. Experiments ran roughly 238,000 model calls across 500 LongMemEval questions, nine commercial models, retrieval-noise conditions, LoCoMo, and a HotpotQA transfer setting.
Results show large, consistent effects. Under matched word budgets, resolved packets outperform recency-truncated raw dialogue by 42–73 points. In deployed-style templates the best-to-worst spread per model reaches 25–49 points. ChatGPT-style natural-language entries produce higher scores than raw conversation on seven of nine models, while formal ledger packets cause three models to score zero even though the same facts are answered at 45–53 percent from natural-language versions. The pattern persists under retrieval noise and transfers to the non-conversational HotpotQA setting.
These findings indicate that memory and RAG benchmarks should either report the reader-facing artifact or include a fixed-artifact control, because otherwise measured gains can reflect differences in evidence rendering rather than differences in model capability or retrieval quality.
Why it matters
Directly actionable for Dutch researchers and advanced practitioners building or evaluating memory/RAG systems; aligns with NL/EU emphasis on transparent, reproducible, and ethical AI evaluation practices.
