Multimodal Entity Linking (MEL) aims to link mentions in a given context to their corresponding entities in a knowledge base by leveraging both textual and visual information. Existing MEL methods still underexploit visual information in real-world scenarios. Most approaches directly encode raw image-text inputs or apply uniform enhancement, while lacking adaptive modeling of heterogeneous mention-relevant evidence. Moreover, they typically rely on single-layer visual representations or simple cross-modal interactions, failing to capture hierarchical visual semantics required for fine-grained disambiguation. To address these limitations, we propose a novel MEL framework termed EVEMEL. First, we introduce an adaptive evidence augmentation mechanism, which analyzes the visual condition of each sample and strengthens mention-relevant discriminative clues, providing more reliable contextual evidence for entity disambiguation. Subsequently, we develop a cross-modal modeling mechanism that treats multi-layer visual representations as different semantic granularities and dynamically fuses them through token-level routing, enabling cross-modal matching. In addition, a bidirectional gated intra-modal mechanism is designed to capture multi-granularity textual and visual matching signals. Experimental results on public benchmark datasets demonstrate that the proposed EVEMEL outperforms all state-of-the-art baselines. The code is publicly available at https://github.com/hanwang86/EVEMEL.