Audio-Visual Speech Recognition (AVSR) systems typically consume only the final-layer representation of their modality-specific encoders. In self-supervised (SSL) encoders, lower and upper layers encode acoustic and more abstract linguistic cues respectively, so the most informative layer may differ from token to token. We propose Mixture of Layer-wise Tokens (MoLT), which aggregates the per-layer outputs of the AV-HuBERT encoder per token through an MoE-style router. A sparse top-k MoLT (S-MoLT) selects the most useful layers per token, and Dynamic routing-based MoLT (D-MoLT) replaces the fixed k with a learnable per-token threshold. To align routing with recognition difficulty, we further add a token-difficulty-aware lower-upper preference that biases, rather than overrides, the learned routing: Difficulty-adaptive D-MoLT (DD-MoLT) derives the difficulty signal from one modality, while Cross-modal DD-MoLT (CDD-MoLT) uses the audio-video reliability gap. On LRS3, our models consistently improve WER over a strong MMS-LLaMA baseline across SNR conditions, with the gain widening under severe noise (e.g., 13.85 vs. 17.10% at−15 dB), at negligible overhead.