Knowledge selection is a crucial sub-task of Document Grounded Dialogue System. Existing methods view knowledge selection as a sentence matching or classification. However, those methods can’t capture the semantic relationships within complex document. We propose a flexible method that can construct multi-level document semantic graph from the grounding document automatically and store semantic relationships within the documents effectively. Besides, we also devise an auxiliary task to leverage the graph more efficiently and can help the optimization of knowledge selection task. We conduct extensive experiments on public datasets: WoW(CITATION) and Holl-E(CITATION). And we achieves state-of-the-art result on WoW. Our code has been released at https://github.com/ddf62/multi-level-semantic-document-graph.