OctoLong: Cross-Repository Code Contexts Enhance Long-Context Modeling

A new study introduces OctoLong, a groundbreaking pipeline for context engineering that uses AST parsing, language server backends, and package managers to gather code references in a recursive manner. This method creates rich code contexts that include millions of tokens and addresses the challenge of limited long-distance dependencies in existing datasets, which mainly contain books, academic articles, and code repositories. The researchers utilized OctoLong to create OctoLong-Instruct, a series of long-context open language models based on base models with parameters between 600M and 14B. The training process involved extending the context mid-training on a mixture of about 50 billion tokens, incorporating approximately 6.2 billion tokens from OctoLong code contexts, plus around 10 billion tokens for instruction tuning. The paper is available on arXiv under the identifier 2608.05141.