The proliferation of large-scale language models has significantly advanced natural language processing capabilities; yet their substantial computational requirements create barriers for deployment in resource-constrained environments and limit accessibility for under-represented language communities. To address this fundamental issue, we present a novel multi-objective pre-training approach for developing parameter-efficient language models. Our method, implemented in the ARJUN model family, strategically combines four complementary pre-training objectives within a unified transformer encoder-decoder framework: (1) Masked Token Prediction (MT) for bidirectional understanding, (2) Next Token Prediction (NT) for autoregressive generation, (3) Masked Span Prediction (MS) for multi-token dependencies, and (4) Complete Sentence Prediction (CS) to bridge discriminative and generative tasks. Using this approach, our ARJUN-Base model (69M parameters) consistently outperforms standard baselines 2.5-4x larger. Extending this architecture, our scaled-up ARJUN-Large variant (up to 256M parameters) demonstrates superior architectural efficiency, establishing strong performance baselines within matched parameter budgets. Evaluations across 20 discriminative and generative tasks in Bangla, Tamil, and Hindi show prominent gains, including up to +6.5 percentage points (pp) over comparable baselines on macro-F1 scores and +17.5 pp sacreBLEU on generation. Finally, ablation studies confirm that joint training on these four objectives yields synergistic representations that consistently outperform single-task configurations across all scales.

Despite remarkable advances in natural language processing, developing effective systems for low-resource languages remains a formidable challenge, with performances typically lagging far behind high-resource counterparts due to data scarcity and insufficient linguistic resources. Cross-lingual knowledge transfer has emerged as a promising approach to address this challenge by leveraging resources from high-resource languages. In this paper, we investigate methods for transferring linguistic knowledge from high-resource languages to low-resource languages, where the number of labeled training instances is in hundreds. We focus on sentence-level and word-level tasks. We introduce a novel method, GETR (Graph-Enhanced Token Representation) for cross-lingual knowledge transfer along with two adopted baselines (a) augmentation in hidden layers and (b) token embedding transfer through token translation. Experimental results demonstrate that our GNN-based approach significantly outperforms existing multilingual and cross-lingual baseline methods, achieving 13 percentage point improvements on truly low-resource languages (Mizo, Khasi) for POS tagging, and 20 and 27 percentage point improvements in macro-F1 on simulated low-resource languages (Marathi, Bangla, Malayalam) across sentiment classification and NER tasks respectively. We also present a detailed analysis of the transfer mechanisms and identify key factors that contribute to successful knowledge transfer in this linguistic context.