Badal Nyalang

Author directory

2026

KrenTransV0-1 is MWire Labs’ submission to the WMT 2026 Low-Resource Indic MT Shared Task. We achieve #1 primary rank on both English→Khasi and English→Kokborok. The system covers all 11 Northeast Indian language pairs using a hybrid architecture combining fine-tuned IndicTrans2 for Indic-script languages, NLLB-200 1.3B mono fine-tuned models for Latin-script languages, and M2M-100 418M fine-tuned models for three severely data-starved languages that exhibited complete output collapse under NLLB. All submissions are primary, covering the English-to-target direction only. We additionally report a negative reranking result: neither BharatGen PARAM-1 (2.9B) perplexity reranking nor NE-BERT fluency reranking improved over a beam-search baseline, and all final submissions use the baseline decoding configuration.
Language identification (LID) is crucial for natural language processing systems, yet Northeast Indian languages remain severely underserved by existing multilingual LID models. We present NE-LID, a fast and accurate language identification system specifically designed for eleven languages of Northeast India. Built using character n-gram features with fastText, NE-LID achieves 99.09% accuracy on a balanced test set, significantly outperforming existing multilingual systems including GlotLID (73.12%), OpenLID (42.03%), IndicLID (39.30%), and LangDetect (24.33%). Our model processes predictions in 0.084 milliseconds on average, enabling real-time applications. We demonstrate that character-level modeling outperforms transformer-based approaches for script-diverse, low-resource languages
Large language models (LLMs) perpetuate cultural stereotypes not only through biased associations but through systematic omission and orthographic erasure of underrepresented languages. We present empirical evidence of two compounding failure modes affecting Northeast Indian languages: (1) entity-level invisibility, where state-of-the-art NER systems score F1=0.000 on culturally critical named entities such as Khasi surnames, Garo festivals, and tribal names; and (2) orthographic corruption, where LLM tokenizers corrupt semantically meaningful diacritics (ï, ñ) and the Garo morpheme boundary marker (U+00B7) at rates of 18.8–50% across four of five evaluated models. Drawing on NortheastNER (F1=0.964, six entity categories, XLM-RoBERTa-base) and a systematic tokenization study across Khasi and Garo, we argue that stereotype-by-omission constitutes a distinct and measurable harm to indigenous language communities. We further show that a custom multilingual tokenizer achieves 26–50% token reduction over five baseline LLMs, demonstrating that culturally grounded infrastructure can partially remediate these failures. Our findings call for cultural representation audits as a standard component of multilingual NLP evaluation.
We present MeiteiRoBERTa, the first publicly available monolingual RoBERTa-based language model for Meitei (Manipuri), a low-resource language spoken by over 1.8 million people in Northeast India. Trained from scratch on 76 million words of Meitei text in Bengali script, our model achieves a perplexity of 65.89, representing a 5.2× improvement over multilingual baselines BERT (341.56) and MuRIL (355.65). Through comprehensive evaluation on perplexity, tokenization efficiency, and semantic representation quality, we demonstrate that domain-specific pre training significantly outperforms general-purpose multilingual models for low-resource languages. Our model exhibits superior semantic understanding with 0.769 similarity separation compared to 0.035 for mBERT and near-zero for MuRIL, despite MuRIL’s better tokenization efficiency (fertility: 3.29 vs. 4.65). We publicly release the model, training code, and datasets to accelerate NLP research for Meitei and other underrepresented Northeast Indian languages
Large pretrained language models have demonstrated remarkable capabilities across diverse languages, yet critically underrepresented low-resource languages remain marginalized. We present NE-BERT, a domain-specific multilingual encoder model trained on approximately 8.3 million sentences spanning 9 Northeast Indian languages and 2 anchor languages (Hindi, English), a linguistically diverse region with minimal representation in existing multilingual models. By employing weighted data sampling and a custom SentencePiece Unigram tokenizer, NE-BERT outperforms IndicBERT-V2 and MuRIL across all 9 Northeast Indian languages, achieving 15.97× and 7.64× lower average perplexity respectively, with 1.50× better tokenization fertility than mBERT. We address critical vocabulary fragmentation issues in extremely low-resource languages such as Pnar (1,002 sentences) and Kokborok (2,463 sentences) through aggressive upsampling strategies. Downstream evaluation on part-of-speech tagging validates practical utility on three Northeast Indian languages. We release NE-BERT, test sets, and training corpus under CC-BY-4.0 to support NLP research and digital inclusion for Northeast Indian communities.
NE-MultiSpeech is a large-scale transcribed multilingual speech corpus covering eight indigenous and regional languages of Northeast India: Khasi, Garo, Mizo, Nagamese, Kokborok, Assamese, Wancho, and Chakma. The corpus combines 124 hours of newly collected, speaker-compensated proprietary recordings across five languages with curated subsets from the Vaani dataset (Pulikodan et al., 2026), totalling approximately 267 hours of transcribed speech across three language families. We fine-tune Whisper-medium on NE-MultiSpeech and establish a systematic automatic speech recognition (ASR) benchmark across all eight languages, achieving an overall word error rate (WER) of 35.53% over 8,099 test utterances jointly, outperforming both Whisper-large-v3 zero-shot (136–194% WER) and MMS-1B zero-shot (42–116% WER) by a large margin. We validate our proxy-language-token strategy for unsupported languages with per-language ablations on four Tibeto-Burman languages, confirming that family-matched proxies consistently outperform mismatched alternatives. We additionally introduce NE-SpeechEmbed, a dual-encoder speech-text retrieval model trained on 73,476 speech-text pairs via InfoNCE contrastive learning, evaluated on the full candidate pool (~1,000 per language) and outperforming CLAP zero-shot across all eight languages and all metrics. Probing the fine-tuned encoder with t-SNE reveals language-family-aligned clustering tendencies without explicit family supervision, supported by a quantitative silhouette score (0.44 language-level, 0.11 family-level). All models, data splits, and utterance IDs are released under CC-BY-4.0; proprietary audio is freely available via an automated access request, used only for usage tracking.
Northeast India is home to over 220 indigenous languages, yet none are served by existing agentic AI systems. NE-Agent addresses this gap: a terminal-based agentic system that integrates the NE-Stack, a suite of purpose-built language AI components for Northeast India, into a unified conversational pipeline that dynamically selects between retrieval, translation, and transcription tools based on the user’s query. The system combines language identification across 11 languages (NE-LID), multilingual dense retrieval (NE-Embed, a LaBSE-based model fine-tuned for 10 Northeast Indian languages), Khasi-to-English translation (fine-tuned NLLB), speech transcription across eight languages (NE-ASR), and generation via a locally-hosted open-source LLM, requiring no API keys or cloud infrastructure. A lightweight LLM-based tool router dispatches each query to the appropriate tool, distinguishing NE-Agent from purely retrieval-augmented systems. NE-Agent is open-source, pip-installable (pip install ne-agent), and ships with a multilingual corpus spanning four languages. To our knowledge, it is the first publicly available agentic system built on indigenous Northeast Indian language AI infrastructure: https://github.com/MWirelabs/ne-agent.