Rounding in Preconditioner Space: Redesigning 4-bit AdamW Optimizer-State Quantization
By Hanyang Li · Paper · cs.LG
Quantizing AdamW's optimizer states reduces persistent storage, but quantization errors propagate through the moment recurrences and perturb subsequent adaptive updates. We redesign 4-bit optimizer-state quantization for AdamW from the perspective of \emph{rounding space}: the co
Tags: Cs.lg