When Attention Goes Blind: Numerical Failure in ALiBi Positional Encodings
By Christopher Schröder · Paper · cs.CL
We identify a previously overlooked failure mode of ALiBi positional encoding: its linear bias scaling underflows floating-point precision, which zeroes out a large fraction of attention weights and renders the affected attention heads partially blind. We analyze this failure mod