Medical VLMs frequently hallucinate clinically unsupported findings. We identify a stability gap: visually grounded tokens maintain relatively stable logits under semantics-preserving image perturbations, whereas visually misgrounded tokens exhibit substantially larger shifts. We introduce LENS (Logit-ENhanced Stability), which exploits this signal at both training and inference. During training, Semantics-Preserving Preference Optimization (SPPO) scores preference pairs using clinical faithfulness and cross-view stability; at inference, stability-aware contrastive decoding penalizes tokens whose logits diverge across original and counterfactual views. Constrained counterfactuals use attribution-guided masking and feature-space proximity guards to reduce disruption of lesion- and device-level evidence. Across radiology VQA, chest X-ray reporting, and gastrointestinal endoscopy, LENS consistently outperforms strong decoding-time and preference-optimization baselines. On MIMIC-CXR, entity-level RadGraph recall improves from 21.0 to 22.8 over PCCD, while expert evaluation shows a hallucination-rate reduction from 17.0% to 8.5%.

Modelling a language model for a multi-lingual scenario includes several potential challenges, among which catastrophic forgetting is the major challenge. For example, small language models (SLM) built for low-resource languages by adapting large language models (LLMs) pose the challenge of catastrophic forgetting. This work proposes to employ a continual learning strategy using parts-of-speech (POS)-based code-switching along with a replay adapter strategy to mitigate the identified gap of catastrophic forgetting while training SLM from LLM. Experiments conducted on vision language tasks such as visual question answering and language modelling task exhibits the success of the proposed architecture.