Transforming Rank: How Architecture Navigates the Spectral Pathologies of Depth
By Katie Everett · Paper · cs.LG
We investigate how each component of the Transformer feedforward block architecture design determines how much rank survives across depth at initialization. We reinterpret skip connections and normalization, long understood as controlling magnitude, as mechanisms for preserving g