DiScoFormer: One transformer for density and score, across distributions

Summary

DiScoFormer is a transformer architecture that recovers both the density and the score of an underlying distribution from a finite sample in one forward pass. Density corresponds to the smoothed probability mass at any location, while the score is the gradient of its logarithm and therefore indicates the direction of steepest increase in probability. The model ingests a set of observed points as context and uses stacked transformer blocks with cross-attention to evaluate these quantities at arbitrary query locations, not merely at the training points.

A single shared backbone feeds two lightweight heads, one producing density values and the other producing scores. Because the score must equal the gradient of the log-density, any discrepancy between the heads supplies a label-free consistency loss. At inference time this loss can be minimized for a few gradient steps while the context remains fixed, allowing the model to adapt on the fly to distributions outside its training support without requiring ground-truth labels.

The architecture is deliberately constructed as a strict generalization of classical kernel density estimation. A single attention head reproduces a Gaussian kernel whose bandwidth is learned rather than fixed, so the network contains KDE as a special case while also learning multiple adaptive scales. Training relies on freshly sampled Gaussian mixture models for every batch; because GMMs admit closed-form densities and scores, the model receives exact supervision across an effectively unlimited family of target distributions.

Empirical comparisons show consistent gains over tuned KDE baselines, with the margin widening in high dimensions. In one hundred dimensions the transformer reduces score error by roughly 6.5 times and density error by more than 37 times relative to the strongest hand-tuned kernel estimator, continues to improve with additional samples where KDE exhausts memory, and generalizes to Laplace, Student-t, and mixtures containing more modes than those seen during training. The resulting pretrained estimator therefore offers a reusable component for diffusion-based generation, Bayesian sampling, and particle simulations across scientific domains.

Why it matters

This article is highly relevant for ML Engineers as it provides a deep dive into a new architectural approach for density and score estimation, crucial for diffusion models and scientific computing. It offers actionable insights into overcoming the high-dimensional limitations of KDE with quantitative benchmarks, making it a valuable tool for Dutch AI teams working on advanced generative AI.