Prefix Sliding for efficient test-time scaling
By Niklas Muennighoff · Paper · cs.CL
Test-time scaling uses extra test-time compute to improve performance, such as letting language models reason longer when solving a problem. As models keep the entire reasoning trace in memory via full attention, hard tasks that need long thinking can be prohibitively expensive.