HEXAGONbeta

Propagation and Optimization of Low-Frequency Spectral Alignment in ReLU Networks on a Fixed Sample Graph

Primarily AI-generated textHuman understanding: some partscs.CG — Computational Geometry

Contributed by Cameron Ryan ↗

Version 1 / Nov 10, 2026 / CC BY 4.0

Abstract

We study how low-frequency structure, measured with respect to the Laplacian of a fixed graph built on the training samples, propagates through a ReLU network to the least-squares readout gg, and how individual neurons h=SΦθh=S\Phi\theta can be optimized to capture low-frequency eigenvectors. The readout error splits into a low-frequency part controlled by coverage (how well the feature span contains the low eigenvectors) and a high-frequency part controlled by the smoothness (Dirichlet energy) of the target. We prove an error bound that makes this precise, with coverage measured by the coverage gap, the largest principal-angle sine between the low eigenvectors and the feature span, and show that coverage can always be preserved across ReLU layers. Finally, we define a count NKN_K of captured low-frequency features, relate it to our error bounds, and prove that error can be reduced to a smoothness component when NKN_K captures every low frequency feature. Lastly, we show that a ReLU layer can strictly increase NKN_K whenever the previous layer keeps the samples distinct and coverage is not yet full, that 2K+22K+2 neurons suffice to preserve coverage, that greedily chosen neurons drive the coverage gap down at rate O(1/t)O(1/t) governed by a ReLU variation norm of the low eigenvectors, and that tt neurons can reduce the coverage gap no further than the (t+1)(t+1)-st principal-angle sine.

Provenance statement

This work was produced using Claude Opus 5.5 with the effort set to high. The theorems, lemmas, corollaries, and propositions were carefully reviewed, and their usefulness was considered. Many edits and changes were made to the paper through an iterative process of iteratively adding and removing sections based on curating useful results. Most important proofs were checked.

Tools used

Anthropic
Claude OpusVersion 5.5 High

References

  1. D. A. Spielman. Spectral and Algebraic Graph Theory . Book draft, Yale University, 2019.
  2. F. R. K. Chung. Spectral Graph Theory , volume 92 of CBMS Regional Conference Series in Mathematics . American Mathematical Society, 1997.
  3. U. von Luxburg. A tutorial on spectral clustering. Statistics and Computing , 17(4):395--416, 2007.
  4. R. A. Horn and C. R. Johnson. Matrix Analysis . Cambridge University Press, 2nd edition, 2013.
  5. G. H. Golub and C. F. Van Loan. Matrix Computations . Johns Hopkins University Press, 4th edition, 2013.
  6. Å. Bj ö rck and G. H. Golub. Numerical methods for computing angles between linear subspaces. Mathematics of Computation , 27(123):579--594, 1973.
  7. G. W. Stewart and J.-G. Sun. Matrix Perturbation Theory . Academic Press, 1990.
  8. C. Davis and W. M. Kahan. The rotation of eigenvectors by a perturbation. III . SIAM Journal on Numerical Analysis , 7(1):1--46, 1970.
  9. P. Drineas, M. Magdon-Ismail, M. W. Mahoney, and D. P. Woodruff. Fast approximation of matrix coherence and statistical leverage. Journal of Machine Learning Research , 13:3475--3506, 2012.
  10. T. M. Cover. Geometrical and statistical properties of systems of linear inequalities with applications in pattern recognition. IEEE Transactions on Electronic Computers , EC-14(3):326--334, 1965.
Show all 19 referencesShow fewer references
  1. M. Pilanci and T. Ergen. Neural networks are convex regularizers: Exact polynomial-time convex optimization formulations for two-layer networks. In Proceedings of the 37th International Conference on Machine Learning (ICML) , 2020.
  2. A. Seeger. Eigenvalue analysis of equilibrium processes defined by linear complementarity conditions. Linear Algebra and its Applications , 292(1--3):1--14, 1999.
  3. E. B. Davies, G. M. L. Gladwell, J. Leydold, and P. F. Stadler. Discrete nodal domain theorems. Linear Algebra and its Applications , 336:51--60, 2001.
  4. C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals. Understanding deep learning requires rethinking generalization. In International Conference on Learning Representations (ICLR) , 2017.
  5. L. K. Jones. A simple lemma on greedy approximation in H ilbert space and convergence rates for projection pursuit regression and neural network training. The Annals of Statistics , 20(1):608--613, 1992.
  6. A. R. Barron. Universal approximation bounds for superpositions of a sigmoidal function. IEEE Transactions on Information Theory , 39(3):930--945, 1993.
  7. R. A. DeVore and V. N. Temlyakov. Some remarks on greedy algorithms. Advances in Computational Mathematics , 5:173--187, 1996.
  8. K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 770--778, 2016.
  9. G. Huang, Z. Liu, L. van der Maaten, and K. Q. Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 4700--4708, 2017.

Version history

  1. v1 · 9 pagesSubmitted by Cameron RyanInitial depositCurrentNov 09, 2026