Audio large language models (LLMs) have advanced rapidly in general audio understanding, yet consistently underperform on tasks requiring fine-grained acoustic discrimination. We investigate this gap through a systematic probing study, uncovering a phenomenon we term representation reorganization: the LLM’s middle layers construct semantic discriminability at the cost of selectively compressing fine-grained acoustic features such as speaker identity and pitch. Informed by this diagnosis, we propose Overtone, a delayed acoustic fusion architecture that bypasses the identified compression zone. Overtone processes identified audio segments through dual tracks—a semantic track from the LLM’s audio encoder and an acoustic track from a frozen self-supervised speech model—and injects acoustic features via gated cross-attention at post-compression layers. The model learns to autonomously decide when and where re-listening is needed during inference. On MMAU and MMAR, Overtone achieves state-of-the-art results among open-source models at comparable scale, with gains concentrated in speech and music—the domains most reliant on fine-grained acoustic features. Post-training probing confirms recovery of acoustic structure in the reorganized layers.