Salekul Islam
Author directory2026
Systematic Failure Analysis of Open-Weight LLM Annotators for Classification Tasks: Taxonomy, Uncertainty Signals, and Deployment Boundaries
Nahid Hossain | Siam Rahman Karip | Md Faisal Kabir | Swakkhar Shatabda | Salekul Islam
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Nahid Hossain | Siam Rahman Karip | Md Faisal Kabir | Swakkhar Shatabda | Salekul Islam
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Large-scale annotated datasets underpin progress in natural language understanding, yet human annotation remains a persistent bottleneck. Large Language Models promise to bridge this gap, but their reliability as annotators is poorly understood. We evaluate 14 open-weight instruction-tuned models and 5 matched base–instruct pairs across 51,189 instances spanning 10 heterogeneous datasets, revealing that annotation failures are structurally predictable rather than random. We identify three recurring failure modes: Safety Hallucination, Intensity Collapse, and Entailment Bias. Safety hallucination can precede alignment rather than being introduced by it: some base models flag up to 100% of benign text as hate speech. Alignment benefits scale inversely with base model capability, yielding a +21.6pp gain for the weakest base model yet a negligible +0.5pp shift for the strongest. KL divergence between base and instruct distributions is 2.0x higher on incorrect predictions, providing a lightweight instance-level signal for alignment-associated errors. Inter-prompt unanimity achieves 72–94% accuracy at 71–90% coverage on clean tasks, yet can actively degrade performance on tasks dominated by shared systematic bias. We translate these findings into a practical deployment framework with pre-screening diagnostics and tiered quality-routing policies, establishing principled boundaries for when LLM annotation can be trusted and when human intervention remains necessary.
HybridScholar: Personalized Paper Recommendation with Cross Attention-Gated Fusion and Verified Explanation Generation
Joytun Nessa Tonny | Sajidur Rahman | Saddam Mukta | Swakkhar Shatabda | Salekul Islam
Proceedings of the 15th International Joint Conference on Natural Language Processing and the 5th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)
Joytun Nessa Tonny | Sajidur Rahman | Saddam Mukta | Swakkhar Shatabda | Salekul Islam
Proceedings of the 15th International Joint Conference on Natural Language Processing and the 5th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)
The exponential growth of scientific literature has made personalized, factually grounded paper discovery increasingly difficult. Existing systems suffer from limited personalization, cold-start fragility, or hallucinated explanations. We introduce HybridScholar, an integrated architecture combining a CrossAttention-gated fusion of collaborative filtering and RAG learning candidate-specific gate weights per candidate with a Chain-of-Verification (CoVe) pipeline for factually grounded explanations. Evaluated on the Scholar Inbox dataset, HybridScholar substantially outperforms (NDCG@10=0.2991) strong baselines including LightGCN and Scholar-Inbox. Cold-start analysis shows the RAG fallback prevents hard failure for sparse users (NDCG@10=0.283), with cold NDCG rising monotonically as interactions grow. The CoVe pipeline reduces hallucination rate by 23.8%, confirmed by a human evaluation study showing CoVe-revised explanations are preferred by experts on factual accuracy (76.7% vs. 36.0%) and by end-users on helpfulness (78.8%, p<0.0001). Overall, HybridScholar improves ranking quality, cold-start robustness, and factual reliability for large-scale scholarly research discovery, demonstrated on 15,881 users across 239,618 papers.