Overfitting of Spectral Gradient Descent: How Matrix Geometry shapes Generalization and Implicit Bias

In this paper, we study directional convergence and benign overfitting of SpecGD in overparametrized matrix linear classification with corrupted labels. We show that changing matrix geometry alone reverse the relative performance of GD and SpecGD.