Test error of interpolating two-layer networks in the feature-learning regime

Two-layer network $f(x) = \sum_{j=1..m} a_j \sigma(w_j . x)$ trained by gradient descent from small initialization to zero training loss on n samples $y = f_{*}(x) + \mathrm{noise}$ of variance $\sigma_n^2$, $x \sim \mathrm{N}(0, I_d)$ with $I_d$ the d x d identity, in the limit $n, d \to \infty$ with $\psi = n/d$ fixed and $m >> n$. For targets $f_{*}(x) = g(U x)$ with $U$ an r x d matrix and r fixed, compute the limiting test mean-squared error on noisy labels $E(\psi, \sigma_n^2, g)$ at fixed $\psi$, and classify the interpolation by the $\psi \to \infty$ limit of $E$: benign if $E \to \sigma_n^2$, tempered if $E \to c\,\sigma_n^2$ with $1 < c < \infty$, catastrophic if $E$ diverges. The answer is an asymptotic formula, proved or confirmed numerically to high precision.

An exact asymptotic formula for the test error of gradient-trained interpolating two-layer networks in the feature-learning regime, matched by large-scale simulation.