Super Weights in LLMs and the Failure of Selective Training
By Shreyas Subramanian · Paper · cs.LG
Recent work identified Super Weights, individual parameters whose removal degrades model performance by orders of magnitude. We show that this degradation due to pruning Super Weights does not universally apply to all LLMs. Furthermore, if these parameters are so important, Super