Evaluating multimodal large language models (MLLMs) on long-form scientific papers requires measuring not only answer correctness, but also whether answers are supported by traceable cross-modal evidence. Existing answer-only metrics and synthetic “Needle-In-A-Haystack” tests mainly assess answer matching or isolated retrieval, leaving evidence-grounded support under-examined. We introduce “Fish-in-the-Ocean” (FITO), an evidence-chain evaluation lens for native scientific documents, and instantiate it with SIN-Data, a scientific interleaved corpus that preserves text–figure coupling. Built on SIN-Data, SIN-Bench comprises four evidence-auditing tasks: evidence discovery, hypothesis verification, grounded QA, and evidence-anchored synthesis. We further propose “No Evidence, No Score”, which credits predictions only when linked to verifiable anchors and diagnoses evidence quality through matching, relevance, and logic. Experiments on 10 MLLMs show a clear correctness–grounding gap: Gemini-3-pro obtains the best average overall score (0.566), while GPT-5 reaches the highest SIN-QA answer accuracy (0.767) but lower evidence-aligned scores. Code and data are available at https://github.com/IIGROUP/sin-bench .

How to evaluate the coding abilities of Large Language Models (LLMs) remains an open question. We find that existing benchmarks are poorly aligned with real-world code repositories and are insufficient to evaluate the coding abilities of LLMs.To address the knowledge gap, we propose a new benchmark named DevEval, which has three advances. (1) DevEval aligns with real-world repositories in multiple dimensions, e.g., code and dependency distributions. (2) DevEval is annotated by 13 developers and contains comprehensive annotations (e.g., requirements, original repositories, reference code, and reference dependencies). (3) DevEval comprises 1,825 testing samples from 115 repositories, covering 10 popular domains (e.g., Internet, Database). Based on DevEval, we propose repository-level code generation and evaluate 8 popular LLMs on DevEval (e.g., gpt-4, gpt-3.5, StarCoder 2, DeepSeek Coder, CodeLLaMa). Our experiments reveal these LLMs’ coding abilities in real-world code repositories. For example, the highest Pass@1 of gpt-4 only is 53.04% in our experiments. We also analyze LLMs’ failed cases and summarize their shortcomings. We hope DevEval can facilitate the development of LLMs in real code repositories. DevEval, prompts, and LLMs’ predictions have been released.

Transformers are now widely used in code representation, and several recent works further develop tree Transformers to capture the syntactic structure in source code. Specifically, novel tree positional encodings have been proposed to incorporate inductive bias into Transformer.In this work, we propose a novel tree Transformer encoding node positions based on our new description method for tree structures.Technically, local and global soft bias shown in previous works is both introduced as positional encodings of our Transformer model.Our model finally outperforms strong baselines on code summarization and completion tasks across two languages, demonstrating our model’s effectiveness.Besides, extensive experiments and ablation study shows that combining both local and global paradigms is still helpful in improving model performance. We release our code at https://github.com/AwdHanPeng/TreeTransformer.