Privacy-preserving adaptation of pretrained language models requires continual fine-tuning as private client distributions shift, while protecting individual samples under local differential privacy (LDP). In streaming differentially private federated learning, full-rank Low-Rank Adaptation (LoRA) adapters expose many trainable directions to DP noise, and later domains may overwrite directions that remain relevant to earlier domains. We propose Private Rank-Invariant Subspace Memory (PRISM), a fixed-rank LoRA framework that schedules the active plastic rank over the stream and applies shrinkage-gated memory writes to DP-protected aggregate updates. Server-side consolidation stores retained high-SNR directions in an orthogonal memory and remains pure post-processing of LDP updates. On MNLI genre and Sentiment3, PRISM stabilizes adaptation under strict privacy budgets while preserving a strong public initialization. At 𝜖=2 , it reaches 79.8% and 92.3% final average accuracy, compared with 52.3% and 85.0% for full-rank DP-FedSVD. Ablations indicate that rank scheduling accounts for most strict-budget recovery, and shrinkage-gated memory contributes a smaller retention gain and modestly improves over the strongest static low-rank control on MNLI.

On-policy distillation (OPD) has recently emerged as an effective post-training paradigm by providing supervision on student-generated trajectories. However, existing OPD methods for multimodal reasoning usually rely on a static teacher routing, assigning each sample to a single teacher based on modality or task type. This ignores that visual grounding and abstract reasoning may dominate different decoding steps, making a single teacher insufficient for the full trajectory. To this end, H-OPD is proposed as a confidence-aware heterogeneous multi-teacher OPD framework for multimodal reasoning. By verifying the complementarity of heterogeneous teachers in the same reasoning process, H-OPD replaces task or sample level teacher routing with token-level teacher arbitration along the shared student trajectory. H-OPD employs vision-to-language description transfer to enable text-only teachers to access key visual semantics, and uses a confidence-aware arbitration mechanism to dynamically combine vision-language teacher and text-only teachers at each token. Extensive evaluations over 11 widely-used reasoning benchmarks showcase the superior performance of our method.