Zhonghao Li

Author directory

2026

Mixture-of-Experts (MoE) models exhibit striking performance disparities across languages, yet the internal mechanisms driving these gaps remain poorly understood. In this work, we conduct a systematic analysis of expert routing patterns in MoE models, revealing a phenomenon we term *Language Routing Isolation*, in which high- and low-resource languages tend to activate largely disjoint expert sets. Through layer-stratified analysis, we further show that routing patterns first converge and then diverge across model depth. Building on these findings, we propose **RISE** (routing isolation-guided subnetwork enhancement), a framework that identifies and adapts language-specific expert subnetworks. RISE uses a tripartite selection strategy: specificity scores select language-specific experts in shallow and deep layers, while overlap scores select universal experts in middle layers. By training only the selected subnetwork while freezing the rest of the model, RISE improves low-resource performance while preserving other-language capabilities. Across 10 languages, RISE achieves target-language F1 gains of up to 10.85% with minimal cross-lingual degradation.

2025

Large language models (LLMs) have made significant strides in natural language processing by leveraging their ability to comprehend and reason with factual knowledge. However, a significant amount of factual knowledge is stored in structured data, which has unique characteristics not typically encountered in the unstructured texts used for pretraining LLMs. To evaluate the capability of LLMs in handling facts structurally stored, we introduce a benchmark called StructFact, which includes meticulously annotated factual questions, spanning five tasks that reflect the intrinsic properties of structured data. This benchmark aims to delineate the strengths and limitations of LLMs in reasoning with structured data for knowledge-intensive tasks in practical applications. Extensive experiments conducted on 10 common LLMs have yielded several insights, one notable finding being that these models struggle significantly with the heterogeneity of structured data during reasoning.

2024