Do VLMs Read or Rewrite? On Transcription Faithfulness in Vision-Language Models
06:00 · July 27, 2026 · arXiv cs.AI RSS
Vision Language Models (VLMs) are increasingly used in place of traditional OCR pipelines for document understanding. In this paper, we show they do not always act as faithful transcribers: when text is imperfect, they often tend to rewrite it into a more plausible form - a behavior that clean-text OCR benchmarks cannot detect. We introduce FaithC4, a multilingual perturbation benchmark of 1,455 single-page documents (English, Chinese, Korean) with three perturbation families: scramble, random substitution, and visually similar substitution. We use the benchmark to evaluate 15 systems spanning general-purpose VLMs, OCR-specialized VLMs, and traditional OCR pipelines. These three categories differ in WER degradation under perturbation: general-purpose VLMs degrade by up to 4.5 points, OCR-specialized VLMs by 0.2-2 points, and traditional OCR by less than 0.6 points on English. Probing Qwen3-VL-4B layer-by-layer, we identify a consistent pattern: rewriting fires only when a perturbed word's final layer FFN representation stays close to the original encoding; when the representation diverges sufficiently, the model transcribes faithfully. Word length affects rewriting rate: short words (4-6 characters) are rewritten up to 10% of the time, with a sharp cutoff at 8 characters above which rewriting drops to 0%.
Summary
Vision Language Models are increasingly deployed for document understanding in place of dedicated OCR pipelines, yet they exhibit a distinct limitation: when presented with imperfect text, they frequently rewrite it into a more plausible form rather than transcribing the visual input literally. This behavior remains invisible on standard clean-text benchmarks. To expose it, the authors created FaithC4, a controlled benchmark of 1,455 single-page documents in English, Chinese, and Korean. Source text drawn from mC4 was subjected to three families of word-level perturbations—character scrambling that preserves first and last letters, random substitution within the same script, and visually similar character replacement—before being rendered as document images.
Evaluation of fifteen systems across three categories reveals clear differences in robustness. General-purpose VLMs suffer word-error-rate increases of up to 4.5 points on English under perturbation, OCR-specialized VLMs show moderate degradation between 0.2 and 2 points, and traditional OCR pipelines remain largely stable with changes below 0.6 points. The pattern holds across the three languages, although the most damaging perturbation type varies with script. Perturbations also produce non-local effects: corrupting only 5 percent of words can increase errors on the remaining clean text by a factor of five to ten in general-purpose models.
Layer-wise probing of Qwen3-VL-4B identifies an internal trigger for the rewriting behavior. When the final-layer feed-forward representation of a perturbed word remains close to its original encoding, the model tends to output the unperturbed form. Once the representation diverges sufficiently, the model transcribes the perturbed characters faithfully. Word length modulates this tendency: short words of four to six characters are rewritten up to 10 percent of the time, while rewriting drops sharply to zero for words longer than seven characters. These findings indicate that language-model priors can override visual evidence inside current VLMs, with implications for applications that require literal transcription such as legal or medical records.
Why it matters
This research is highly relevant for Dutch AI researchers and enterprises deploying VLMs for document understanding, particularly in sectors requiring strict transcription accuracy like legal, medical, and government digitization. It provides actionable insights into VLM hallucination mechanisms, aligning with EU AI Act requirements for model reliability and transparency.
