OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modeling
By Indraneil Paul · Paper · cs.AI
Context lengths of language models (LMs) have dramatically increased, driven by the demands for in-context learning, self-improvement, and long-horizon agentic workflows. Existing long-context corpora, however, are dominated by books, academic articles, and code repositories, whi