Procedural Knowledge Is Not Low-Rank: Why LoRA Fails to Internalize Multi-Step Procedures
06:00 · July 27, 2026 · arXiv cs.AI RSS
Parameter-efficient fine-tuning methods like LoRA have become the default for adapting large language models, succeeding across instruction following, style transfer, and factual adaptation. We show that for procedural knowledge--the ability to follow multi-step procedures with conditional branching through to terminal states--LoRA fails to match full fine-tuning at the ranks where it retains its efficiency advantage. In a systematic ablation (r = 16--128) on a procedural travel booking task (14 nodes), all LoRA configurations fail uniformly (task success <= 2.54 vs. 4.11 for full fine-tuning, all p < 0.001), with scores decreasing at higher ranks--despite maintaining 95--99% conversation completion rates. Cross-domain replication on Zoom support (14 nodes) and insurance claims (55 nodes) at 8B confirms the failure generalizes: LoRA underperforms full fine-tuning by 0.8--2.2 points on average at both r = 32 and r = 128, with the largest gap on the most complex procedure. Quadrupling rank from 32 to 128 provides marginal improvement but does not close the gap. SVD analysis of the weight changes produced by full fine-tuning explains why: across three domains at both 3B and 8B, the mean effective rank of the update ranges from 761 to 1,026, and rank 128 captures only 43--51% of the squared Frobenius norm. Together, these findings establish that for procedural tasks LoRA falls well short of full fine-tuning--a fundamental limitation for agentic applications.
Summary
The paper shows that low-rank adaptation methods such as LoRA cannot reliably encode procedural knowledge in large language models, even when the rank is raised well beyond typical efficiency settings. Procedural knowledge here refers to the capacity to follow multi-step workflows that include conditional branching, implicit state tracking across dialogue turns, and progression toward defined terminal states. Systematic rank ablations on a travel-booking task with 14 nodes found that LoRA configurations at ranks 16 through 128 all produced task-success scores below 2.54, compared with 4.11 for full fine-tuning, while conversation-completion rates remained above 95 percent. Performance did not improve, and in some cases declined, as rank increased.
Replication on two further domains confirmed the pattern. On an eight-billion-parameter model, LoRA at both rank 32 and rank 128 trailed full fine-tuning by 0.8 to 2.2 points on a Zoom-support workflow and on a substantially larger insurance-claims procedure containing 55 nodes. The performance gap widened with procedure complexity, and simply quadrupling the rank from 32 to 128 yielded only marginal gains that failed to close the difference.
Singular-value decomposition of the weight updates produced by full fine-tuning supplies a structural explanation. Across the three domains and both three- and eight-billion-parameter models, the mean effective rank of the learned updates ranged from 761 to 1,026. A rank-128 matrix therefore captured only 43 to 51 percent of the squared Frobenius norm of the change. Training dynamics further indicate that the shortfall is not an optimization artifact: LoRA reached lower held-out per-token loss than full fine-tuning yet still failed to acquire the required state-to-action mappings. The findings point to an inherent limitation of low-rank parameter updates when the target behavior is a high-dimensional procedural mapping rather than a low-dimensional stylistic or factual shift.
Why it matters
Directly actionable for Dutch AI teams building LLM agents; underscores need for full fine-tuning in procedural domains relevant to EU-regulated enterprise applications. High technical depth and novelty on fine-tuning limitations.
