FLOPs vs Real Work: The Importance of Replication in AI Efficiency Assessment
06:00 · August 18, 2026 · arXiv cs.AI RSS
AI efficiency has recently taken the spotlight in both academy and industry due to massive model scales, high energy demands, and environmental costs. While reporting Floating Point Operations (FLOPs) is a traditional approach for assessing computational costs, the relationship between FLOPs and execution time is not straightforward, as layers with the same number of FLOPs may not have the same execution time because some operations are more easily parallelized than others. This paper sets out to replicate the original experiments from a study that proposed the $\alpha-FLOPs$ estimation formula to verify whether the results remain applicable on newer, more powerful hardware. During the replication process, we identify limitations in the replication materials provided by the original study, including a lack of specific dependency details and transparency regarding regression data. Our results validate the thesis that raw FLOPs alone are not an appropriate metric for execution time, as spatial dimensions remain more easily parallelized than kernel dimensions. However, fine-grained measurements reveal that the relationship is much less straightforward than previously shown, with newer hardware exhibiting instabilities and discontinuities in execution time, including jumps and oscillations, that the $\alpha-FLOPs$ formula generally underestimates. Ultimately, this work validates the empirical findings from the original study but shows negative results when applying the $\alpha-FLOPs$ estimation. We also highlight the critical need for complete and accurate replication packages for research on hardware-dependent efficiency assessment and provide a complete replication package for our implementation to facilitate further study.
Summary
A replication study by researchers at TU Delft revisits earlier work on measuring the computational cost of convolutional neural networks. The authors test whether counting raw floating-point operations remains a reliable predictor of execution time when models run on contemporary hardware such as the RTX 4090. Their experiments confirm that layers with identical theoretical FLOP counts can still differ markedly in runtime, because operations along spatial dimensions parallelize more readily than those involving kernel size or channel depth.
Fine-grained timing on the newer accelerator reveals behavior that the original study did not capture. Execution times exhibit instabilities, abrupt jumps, and oscillations that are absent from coarser measurements. As a result, the α-FLOPs correction formula derived from earlier regression data consistently underestimates observed runtimes. While the broad empirical claim—that raw FLOP counts alone are insufficient—holds, the quantitative adjustment proposed by the prior work does not transfer directly.
The replication effort also exposes shortcomings in the materials supplied by the original authors. Missing dependency specifications, undisclosed regression data, and incomplete experiment scripts hindered faithful reproduction. In response, the TU Delft team released a complete, self-contained replication package that includes both measurement code and analysis scripts, underscoring the importance of transparent artifacts when efficiency claims depend on specific hardware characteristics.
Why it matters
Directly relevant for Dutch AI researchers and advanced practitioners working on Green AI, model optimization, and reproducible efficiency metrics; authors are local, findings address EU energy concerns, and results are actionable for accurate cost assessment on modern GPUs.
