SemHash-LLM: A Multi-Granularity Semantic Hashing Framework for Document Deduplication
06:00 · July 3, 2026 · arXiv cs.AI RSS
Large scale document deduplication must preserve semantic equivalence while remaining efficient over massive corpora. We present SemHash LLM, a multi granularity framework that unifies semantic projection hashing, attention weighted MinHash, contrastive boundary learning, and selective LLM based adjudication. The method combines character, token, and document level signals through gated fusion, then applies a cascaded filtering pipeline for efficient candidate reduction. Semantic projection hashing learns compact binary codes in distilled LLM embedding space, while attention weighted Min- Hash suppresses boilerplate and emphasizes informative content. Adaptive decision boundaries and uncertainty estimation further improve robustness across template pollution, short text perturbation, containment, and viral fragments. Experiments show that SemHash LLM achieves strong duplicate detection quality with less than one percent neural verification cost.
Summary
SemHash-LLM addresses the tension in large-scale document deduplication between preserving semantic equivalence and controlling computational cost across heterogeneous web corpora. Traditional lexical fingerprinting remains fast yet brittle under paraphrase or template wrapping, while full embedding-based approaches incur prohibitive verification overhead when applied to billions of documents. The framework resolves this by integrating signals at character, token, and document levels through a gated fusion network, then routing candidates through a cascaded pipeline of Bloom filters, semantic hash blocking, and attention-weighted locality-sensitive hashing.
At its core, semantic projection hashing distills a compact student encoder from a larger LLM teacher to produce embeddings that are subsequently mapped to binary codes via learned hyperplane partitions. Attention-weighted MinHash further refines candidate generation by deriving importance weights from transformer attention patterns, thereby down-weighting boilerplate while emphasizing informative content. Adaptive decision boundaries learned through contrastive objectives, combined with uncertainty estimation, improve robustness across template pollution, short-text perturbations, parent-child containment, and high-frequency viral fragments.
Only pairs that remain ambiguous after these automated stages are forwarded to an LLM-as-judge module, which supplies structured adjudication fused with the preceding predictions. This selective routing keeps neural verification costs below one percent of the corpus while preserving duplicate detection quality comparable to more expensive semantic baselines. The design therefore supports trillion-scale curation pipelines without forcing a binary choice between lexical speed and semantic fidelity.
Why it matters
This research is highly relevant for Dutch AI researchers and engineers building large-scale NLP pipelines or training datasets, as efficient deduplication reduces computational overhead and improves data quality. The techniques align with EU goals for resource-efficient and high-quality AI development.
