A Positive Dispersion Optimum for the Rounded Sample Mean: Sufficiency and Necessity of Finite Variance

A mix of human-written and AI-generated textHuman understanding: all partsmath.ST — Statistics Theory

Contributed by Mark R. Diamond ↗, Angela O'Brien-Malone ↗

Version 1 / Oct 14, 2026 / CC BY 4.0

Abstract

A continuous quantity, observable only after nearest-lattice rounding, is estimated by the mean of nn rounded readings of an imperfect instrument. Averaged over the position of the true value within a rounding cell, the mean-squared error of this estimator is a function of one dimensionless parameter: the pre-rounding standard deviation ss, measured in cell widths. A perfectly repeatable instrument (s=0s=0) returns the same rounded value on every reading, so averaging provides no additional information; its cell-averaged error is the rounding floor 1/121/12. We show that this floor is never optimal once more than one reading is taken. For \emph{any} centred pre-rounding law with finite, non-zero variance and any sample size n>1n>1, some strictly positive dispersion yields a smaller cell-averaged error than zero, so the global minimizer lies away from the no-scatter endpoint; and each of the three hypotheses---n>1n>1, finite variance, non-degeneracy---is individually necessary, so existence is \emph{exactly} characterized. The mechanism is elementary: the autocorrelation of the rounding error has a \emph{corner} at zero lag, so a little scatter relieves the rounding bias at first order in ss while its variance contribution grows only at second order. We give the short corner-based proof and then derive the spectral form of the criterion, using it to compute the optimum for four laws. The Gaussian optimizer is unique and solves a Jacobi-theta equation. For the Laplace law the bias term has a hyperbolic closed form, and the optimizer is unique and characterized by the corresponding derivative equation. For the uniform and two-point laws the optimizer is explicit. The uniform law cancels every bias harmonic exactly at its first positive cancellation scale, which saturates the natural localization bound when n=2n=2; and the two-point law, whose criterion has several local wells once n≥4n\ge4, shows that finite variance ensures existence but not a single-well objective.

Provenance statement

The problem considered in this paper has its origin in Lumsden's~\cite{Lumsden1976} ``flogging wall'' thought experiment, first encountered by MRD as an undergraduate student in 1979. Lumsden used the flogging wall to test the counterintuitive possibility that, in some circumstances, an unreliable psychometric instrument might outperform a reliable one. Between 1989 and 2014, Daniel D. Reidpath (ORCID: \href{https://orcid.org/0000-0002-8796-0420}{0000-0002-8796-0420}) and MRD intermittently explored a variety of formalized simulation models of the flogging wall which strongly suggested that instrument unreliability could be advantageous. Two unpublished papers were developed on the basis of the simulation results. In 2023 MRD reformulated the problem in terms of a weighing scale that \emph{truncates} to finite precision, as opposed to the simpler rounding formulation used here. AOBM observed that this finite precision formulation could be expressed naturally as a question about quantization; see for example \cite[p.~1]{WidrowKollar2008}. Our first formalization was restricted to Gaussian scatter using, as a starting point, the treatment of quantized distributions given by Schwartz \cite{Schwartz1980}, and drawing also on the quantization framework of Widrow~\cite{Widrow1961}. This led to a moment-based derivation of a formula for the mean-squared error \(\MSE_n(u,s)\) around a specific $u$, followed by integration over \(u\in[-\tfrac12,\tfrac12)\), differentiation with respect to \(s\), and a Jacobi theta-function expression for the minimizing scale. That work was developed into a short unpublished manuscript. The present paper arose from the attempt to identify which features of the Gaussian calculation were essential. In particular, MRD conjectured from the moment-based model that positive finite variance might be the necessary and sufficient condition for the existence of a strictly positive minimizer of the cell-averaged mean-squared error, for \(n>1\). The central simplification in the present paper is the replacement of the Gaussian moment calculation by the autocorrelation formulation used above. This change separates the existence argument from the special analytic form of the Gaussian density, and is what allows the result to be stated for the more general class of distributions considered here. Figures were initially developed in Wolfram Mathematica and then translated into TikZ for inclusion in the manuscript. The mathematical arguments, exposition, and final form of the paper are the responsibility of the authors. \section*{Acknowledgements and AI tool disclosure} The authors used several generative AI tools during the development of this paper, including OpenAI ChatGPT, Anthropic Claude, Google Gemini, and Moonshot AI Kimi. The tools were used to check algebraic derivations from the quantization literature, to test and expand justifications involving Fourier series, convergence, and exchanges of limits, sums, and integrals, and to help identify simpler proof strategies from earlier moment-based formulations. In particular, ChatGPT and Claude independently suggested that it might be possible to simplify the existence argument by working with an autocorrelation function rather than with the moment formula derived from the Gaussian model. AI tools were also used for proofreading and for checks of notational consistency. Google Gemini and Moonshot Kimi were particularly helpful with LaTeX assistance, and the translation of Wolfram Mathematica plots into TikZ. All AI-assisted mathematical suggestions were reviewed, revised, and checked by the authors. The text, mathematical claims, and final editorial decisions are those of the authors, who take responsibility for any remaining errors.

Tools used

Anthropic
Claude OpusVersion 4
Google
Gemini
OpenAI
ChatGPTVersion 4o
Moonshot
Kimi
Wolfram
MathematicaVersion 14.1

References

  1. A. Chapman. Dividing the Circle: The Development of Critical Angular Measurement in Astronomy, 1500--1850 . 2nd edition, John Wiley & Sons, Chichester, 1995.
  2. A. Hald. On the history of the correction for grouping, 1873--1922. Scandinavian Journal of Statistics , 28(3):417--428, 2001. doi:10.1111/1467-9469.00245 (https://doi.org/10.1111/1467-9469.00245).DOI
  3. S. P. Lipshitz, R. A. Wannamaker, and J. Vanderkooy. Quantization and dither: a theoretical survey. Journal of the Audio Engineering Society , 40(5):355--375, 1992.
  4. J. Lumsden. Test theory. Annual Review of Psychology , 27(1):251--280, 1976.
  5. L. Schuchman. Dither signals and their effect on quantization noise. IEEE Transactions on Communication Technology , COM-12(4):162--165, December 1964. doi:10.1109/TCOM.1964.1088973 (https://doi.org/10.1109/TCOM.1964.1088973).DOI
  6. L. M. Schwartz. Round-off error. Analytical Chemistry , 52(7):1141--47. doi:10.1021/ac50057a033 (https://doi.org/10.1021/ac50057a033).DOI
  7. W. F. Sheppard. On the calculation of the most probable values of frequency-constants, for data arranged according to equidistant division of a scale. Proceedings of the London Mathematical Society , s1-29(1):353--380, 1898. doi:10.1112/plms/s1-29.1.353 (https://doi.org/10.1112/plms/s1-29.1.353).DOI
  8. O. B. Sheynin. Mathematical treatment of astronomical observations: A historical essay. Archive for History of Exact Sciences , 11(2--3):97--126, 1973.
  9. O. B. Sheynin. The true value of a measured constant and the theory of errors. Historia Scientiarum , 17(1):38--48, 2007.
  10. T. Simpson. A letter to the Right Honourable George Earl of Macclesfield, President of the Royal Society, on the advantage of taking the mean of a number of observations, in practical astronomy. Philosophical Transactions of the Royal Society of London , 49:82--93, 1755. doi:10.1098/rstl.1755.0020 (https://doi.org/10.1098/rstl.1755.0020).DOI
Show all 12 referencesShow fewer references
  1. B. Widrow. Statistical analysis of amplitude-quantized sampled-data systems. Transactions of the American Institute of Electrical Engineers, Part II: Applications and Industry , 79(6):555--568, 1961. doi:10.1109/TAI.1961.6371702 (https://doi.org/10.1109/TAI.1961.6371702).DOI
  2. B. Widrow and I. Kollár. Quantization Noise: Roundoff Error in Digital Computation, Signal Processing, Control, and Communications . Cambridge University Press, Cambridge and New York, 2008.

Version history

  1. v1Submitted by Mark R. DiamondInitial depositCurrentOct 14, 2026