cs.CG — Computational Geometry

Propagation and Optimization of Low-Frequency Spectral Alignment in ReLU Networks on a Fixed Sample Graph

Contributed by Cameron Ryan

We study how low-frequency structure, measured with respect to the Laplacian of a fixed graph built on the training samples, propagates through a ReLU network to the least-squares readout $g$, and how individual neurons $h=S\Phi\theta$ can be optimized to capture low-frequency eigenvectors. The readout error splits into a low-frequency part controlled by coverage (how well the feature span contains the low eigenvectors) and a high-frequency part controlled by the smoothness (Dirichlet energy) of the target. We prove an error bound that makes this precise, with coverage measured by the coverage gap, the largest principal-angle sine between the low eigenvectors and the feature span, and show that coverage can always be preserved across ReLU layers. Finally, we define a count $N_K$ of captured low-frequency features, relate it to our error bounds, and prove that error can be reduced to a smoothness component when $N_K$ captures every low frequency feature. Lastly, we show that a ReLU layer can strictly increase $N_K$ whenever the previous layer keeps the samples distinct and coverage is not yet full, that $2K+2$ neurons suffice to preserve coverage, that greedily chosen neurons drive the coverage gap down at rate $O(1/t)$ governed by a ReLU variation norm of the low eigenvectors, and that $t$ neurons can reduce the coverage gap no further than the $(t+1)$-st principal-angle sine.