Vision-Language-Action (VLA) models are built on the premise that pre-trained Vision-Language Models (VLMs) can supply general semantic understanding ability for embodied tasks. However, we find that robotic fine-tuning can severely erode this capability: VQA benchmarks and layer-wise CKA analysis reveal rapid representational drift, and co-training with general vision-language data does not prevent the collapse. We propose TwinBrainVLA, an asymmetric dual-stream VLA that keeps a pre-trained "Left Brain" strictly frozen as a semantic generalist while training a "Right Brain" as an embodied control specialist. Its Asymmetric Mixture-of-Transformers (AsyMoT) mechanism lets the Right Brain query frozen Left-Brain key-value representations at every transformer layer, integrating intact semantic knowledge without corrupting it. Across SimplerEnv, RoboCasa, LIBERO, and real-robot experiments, TwinBrainVLA outperforms strong baselines and large-scale embodied models, achieving a 14.3% improvement on SimplerEnv and a 2.7× gain in physical-robot instruction generalization.