Sample complexity of gradient training for multi-index targets · revision history

Revision 2

Accepted · core change · record

Changes against revision 1:

Title
Sample complexity of gradient training for multi-index targets
Plain
A network trained by stochastic gradient descent (repeated small weight updates using random batches of data) must find a few hidden directions in a high-dimensional input. The question is how many training examples it needs as the input dimension grows, compared with the minimum any fast method needs.
Precise · changed · core
Revision 1: Inputs $x \sim N(0, I_d)$ and labels $y = g(W x)$ with $W$ an $r$ x d matrix of rank r fixed as $d \to \infty$ and $g$ a fixed link function. For a standard two-layer network trained by online or multi-pass SGD on the square loss without data preprocessing, determine the exponent $\gamma(g)$ in $n \sim d^\gamma$ required to reach test error $o(1)$, and determine whether it equals the benchmark for statistical-query and low-degree algorithms, $n \sim d^{\operatorname{max}(1, k_*/2)}$, with $k_*$ the generative exponent for single-index targets and the generative leap exponent for multi-index targets (Damian, Lee and Bruna, arXiv:2506.05500). The answer is $\gamma$ as a function of the Hermite structure of $g$, with proof.
Revision 2: Inputs $x \sim N(0, I_d)$ and labels $y = g(W x)$ with $W$ an $r$ x d matrix of rank r fixed as $d \to \infty$ and $g$ a fixed link function. For a standard two-layer network trained by online or multi-pass SGD on the square loss without data preprocessing, determine the exponent $\gamma(g)$ in $n \sim d^\gamma$ required to reach test error $o(1)$, and determine whether it equals the benchmark for statistical-query and low-degree algorithms, $n \sim d^{\max(1, k_*/2)}$, with $k_*$ the generative exponent for single-index targets and the generative leap exponent for multi-index targets (Damian, Lee and Bruna, arXiv:2506.05500). The answer is $\gamma$ as a function of the Hermite structure of $g$, with proof.
Settled by
Matching upper and lower bounds on the sample complexity of standard gradient training for general multi-index targets.
Kind
well-posed
Answer type
value
References
(empty)
Literature status
partially-resolved
Status note
For single-index polynomial targets, SGD with minibatch reuse on two-layer networks reaches $n \sim d\,\operatorname{polylog} d$ (Lee, Oko, Suzuki and Wu, arXiv:2406.01581, 2024); correlational SGD on multi-index targets is governed by the leap complexity (Abbe, Boix-Adsera and Misiakiewicz 2023); the low-degree benchmark for multi-index targets is set by the generative leap (Damian, Lee and Bruna 2025); whether standard SGD reaches that benchmark for general multi-index $g$ is open (2026).
Posed since
(empty)

Decisions