With the growing use of large language models (LLMs) as evaluators, their application has expanded to code evaluation tasks, where they assess the correctness of generated code without relying on reference implementations. While this offers scalability and flexibility, it also raises a critical, unresolved question: Can LLM judges fairly and robustly evaluate semantically equivalent code with superficial variations? Functionally correct code often exhibits variations—such as differences in variable names, comments, or formatting—that should not influence its correctness. Yet, whether LLM judges can reliably handle these variations remains unclear. We present the first comprehensive study of this issue, defining six types of potential bias in code evaluation and revealing their systematic impact on LLM judges. Across five programming languages and multiple LLMs, we empirically demonstrate that all tested LLM judges are susceptible to both positive and negative biases, resulting in inflated or unfairly low scores. Moreover, we observe that LLM judges remain vulnerable to these biases even when prompted to generate test cases before scoring, highlighting the need for more robust code evaluation method.

As large language models (LLMs) increasingly process instructions from multiple sources with different privilege ranks, preserving the priority of higher-rank directives over conflicting lower-rank ones is critical for safe and predictable deployment. However, existing evaluations mostly focus on limited conflict settings, leaving it unclear how models handle instruction hierarchy (IH) when multiple conflicts arise. To address this gap, we present the first systematic study of IH under multi-conflict settings and introduce MIH-Bench, a benchmark that scales the number of simultaneous conflicts across six pairwise hierarchy scenarios. Experiments on 21 LLMs show that IH compliance deteriorates sharply as conflicts accumulate, while neither model scale nor generation reliably predicts robustness. We further find that system-involving conflicts are the most vulnerable, a particularly concerning result given the safety-critical role of system prompts. Moreover, reasoning can improve conflict resolution, although the gains vary across models and are bounded by underlying model capability. These results show that current LLMs remain fragile in pairwise multi-conflict settings, highlighting the need for stronger hierarchy-aware alignment. Code and data are available at https://anonymous.4open.science/r/MIH-release-A5B8/

Instruction hierarchy (IH) requires models to prioritize instructions by source, ensuring that higher-priority instructions override lower-priority ones. Despite its importance for safe and controllable deployment, existing evaluations have focused almost exclusively on English, leaving it unclear whether IH compliance remains stable in multilingual settings. We introduce XIH-Bench, a benchmark for multilingual IH evaluation with both same-language and cross-language conflicts across six languages, four domains, and three IH settings. Across models, we find two consistent patterns. First, IH compliance exhibits a clear language-dependent asymmetry: a language that strengthens compliance in the higher-priority position can become disruptive in the lower-priority position. Second, cross-language conflicts yield higher compliance than same-language conflicts, a phenomenon we term the Language Boundary Effect. We further show that language specialization can make lower-priority instructions in model-favored languages harder to override, creating multilingual reliability and security risks.