Academic

Abstract

The spiked random matrix is a classical statistical model describing a low-dimensional signal “hidden” in high-dimensional noise. In this talk we introduce two examples where the spiked matrix model provides insight to modern machine learning problems, especially in characterizing the performance of neural networks and kernel models. First we consider the learning of a single-index target function under isotropic Gaussian input. In this setting, it is known that kernel method suffers from the curse of dimensionality in the proportional asymptotic limit; whereas for two-layer neural network, we show that the updated parameters after the first gradient step exhibit a signal (spike) + noise (bulk) structure, which can significantly improve the model performance. In the second example, we consider the scenario where the (anisotropic) input data already contains low-dimensional structure given by a spiked covariance. We show that both kernel ridge regression and neural network benefit from anisotropy, but neural network can adapt to such a structure more effectively.