REER-PT: Reverse-Engineered Reasoning for Perplexity-Guided Pre-training Data Augmentation
By Haoran Que · Paper · cs.CL
As language-model compute continues to scale, high-quality training data is becoming an increasingly important bottleneck. Conventional next-token prediction supervises what follows a context but leaves the intermediate reasoning behind that continuation implicit. We introduce \t