Code reasoning is essential for building AI systems that can understand, generate, and verify complex software. Yet most open-source code corpora focus on isolated snippets or single functions, failing to capture the long-range dependencies and cross-file interactions that define real-world repositories. We introduce MegaCode, a 300B repository-level corpus designed to support structured code reasoning and long-context modeling. MegaCode preserves full project structure—including build scripts, configuration files, and inter-file links—and is constructed through a multi-stage curation pipeline combining deduplication, rule-based filtering, and model-based quality assessment. Models pretrained on MegaCode demonstrate consistent gains on code reasoning and long-context benchmarks compared with those trained on flattened code datasets. Also, we validate the effectiveness of adding repo relevent infomation such as LSP, repo structure and repo summary. These results highlight the importance of maintaining repository structure for next-generation code intelligence. We release the dataset, curation pipeline, and evaluation suite to facilitate research on large-scale code pretraining and repository-level understanding.

As access to high-quality, domain-specific data grows increasingly scarce, multi-epoch training has become a practical strategy for adapting large language models (LLMs). However, autoregressive models often suffer from performance degradation under repeated data exposure, where overfitting leads to a marked decline in model capability. Through empirical analysis, we trace this degradation to an imbalance in learning dynamics: predictable, low-entropy tokens are learned quickly and come to dominate optimization, while the model’s ability to generalize on high-entropy tokens deteriorates with continued training. To address this, we introduce EntroDrop, an entropy-guided token dropout method that functions as structured data regularization. EntroDrop selectively masks low-entropy tokens during training and employs a curriculum schedule to adjust regularization strength in alignment with training progress. Experiments across model scales from 0.6B to 8B parameters show that EntroDrop consistently outperforms standard regularization baselines and maintains robust performance throughout extended multi-epoch training. These findings underscore the importance of aligning regularization with token-level learning dynamics when training on limited data. Our approach offers a promising pathway toward more effective adaptation of LLMs in data-constrained domains.

“CCL2024-Eval任 务7为 中 小 学 生 作 文 流 畅 性 评 价 (Chinese Essay Fluency Evalua-tion,CEFE),该任务定义了三项重要且富有挑战性的问题,包括中小学作文病句类型识别、中小学作文病句改写、以及中小学作文流畅性评级。本队伍参加了评测任务7的三项子任务,分别获得了45.19、43.90和45.84的得分。本报告详细介绍本队伍在三个子任务上采用的技术方法,并对评测结果进行分析。”