Real-world documents are visually rich and structurally organized, with text, tables, figures, and captions interleaved across pages. Although retrieval-augmented generation (RAG) is widely used for document question answering, existing pipelines often treat such documents as fixed page- or chunk-level collections. This document-agnostic construction either dilutes evidence in dense pages or fragments semantically related layout elements. Moreover, cross-page questions often require iterative evidence exploration, but existing systems typically follow fixed retrieval paths that do not adapt to the evidence already found.

We propose DAMG-RAG, a Document-Aware Multi-Granularity RAG framework for visually rich document understanding. During ingestion, DAMG-RAG constructs layout-aware atomic retrieval units and links them to page- and section-level nodes, forming a multi-granularity document tree. During inference, a state-aware agentic workflow maintains explored regions and validated evidence, dynamically selecting exploration and navigation tools over this tree. On MMLongBench, DAMG-RAG obtains 64.7% accuracy with Claude Sonnet 4.6, yielding a 3.6-point gain over the best-performing agentic baseline in our experiments while using 18.8% of its token budget.