Large language models (LLMs) remain challenged by ultra-long document understanding. A common solution is to segment documents into chunks and retrieve those relevant to a query, but this can disrupt semantic and logical coherence by ignoring dependencies between chunks. We propose NEAR (NEighborhood-Aware context Reconstruction), a lightweight, training-free framework that reconstructs coherent context from fragmented segments. NEAR represents each document as a dual-layer neighborhood graph, connecting chunks via topical neighbors to capture long-range semantic relations and physical neighbors to preserve local continuity. An iterative dual-pool exploration traverses chunks from both global and local neighborhoods, and collected chunks are selectively expanded and reordered according to their original positions to form a coherent input context. Extensive experiments on multiple long-document question answering benchmarks demonstrate that NEAR improves comprehension performance while maintaining low latency and memory usage.