Large Language Models (LLMs) are sensitive to the order in which answer choices are presented in Multiple-Choice Question Answering. The same question can yield a different prediction when its options are shuffled. Although exhaustive majority voting over all permutations can mitigate this bias, its factorial cost makes it impractical at inference time. We propose GPS (Guided Permutation Scoring), a two-stage inference-time pipeline that selects option orderings more likely to produce correct predictions from a frozen LLM. A sensitivity classifier first identifies which questions are genuinely vulnerable to positional interference; robust questions are answered directly, while sensitive ones are forwarded to a trained permutation scorer that identifies a more position-stable option ordering, promoting better predictions. Unlike prior approaches that either enumerate all permutations uniformly, or require modifying the LLM’s parameters, GPS is the first method to actively select permutations so that the base model can consistently give better answers. GPS steadily improves Accuracy, and Macro F1 across all evaluated model-dataset combinations, with remarkable reductions in positional bias metrics.

We introduce SHONGLAP, a large annotated open-domain dialogue corpus in Bengali language. Due to unavailability of high-quality dialogue datasets for low-resource languages like Bengali, existing neural open-domain dialogue systems suffer from data scarcity. We propose a framework to prepare large-scale open-domain dialogue datasets from publicly available multi-party discussion podcasts, talk-shows and label them based on weak-supervision techniques which is particularly suitable for low-resource settings. Using this framework, we prepared our corpus, the first reported Bengali open-domain dialogue corpus (7.7k+ fully annotated dialogues in total) which can serve as a strong baseline for future works. Experimental results show that our corpus improves performance of large language models (BanglaBERT) in case of downstream classification tasks during fine-tuning.