<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="cao-2026-policy">
    <titleInfo>
        <title>Policy Loopholes in Agent Evaluation: When Policy Ambiguity Masquerades as Agent Error</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Hongliu</namePart>
        <namePart type="family">Cao</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-10</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the Second Workshop for REsearch on Agent Language Models (REALM 2026)</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Ehsan</namePart>
            <namePart type="family">Kamalloo</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Nicolas</namePart>
            <namePart type="family">Gontier</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Nikolai</namePart>
            <namePart type="family">Rozanov</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Maryam</namePart>
            <namePart type="family">Hashemzadeh</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Akhil</namePart>
            <namePart type="family">Arora</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Budapest, Hungary</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-485-9</identifier>
    </relatedItem>
    <abstract>Agent benchmarks evaluate policy compliance but assume each policy determines a unique correct action. Natural-language policies can violate this assumption through silence, ambiguity, or contradiction, admitting multiple defensible readings that a single gold trajectory cannot capture. Auditing two τ²-bench domains, we develop a taxonomy of such policy loopholes and show that affected tasks produce unreliable scores: they lower scores across different models in different ways and make every model less consistent across repeated trials. A cross-domain comparison reveals that exploitability requires both policy ambiguity and tool permissiveness: when policy complexity exceeds what tools can enforce, agents resolve gaps inconsistently and scores become unreliable. Policy specification quality sets the ceiling on evaluation quality. Benchmark developers should audit policies before collecting gold annotations.</abstract>
    <identifier type="citekey">cao-2026-policy</identifier>
    <location>
        <url>https://aclanthology.org/2026.realm-1.98/</url>
    </location>
    <part>
        <date>2026-10</date>
        <extent unit="page">
            <start>1404</start>
            <end>1419</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Policy Loopholes in Agent Evaluation: When Policy Ambiguity Masquerades as Agent Error
%A Cao, Hongliu
%Y Kamalloo, Ehsan
%Y Gontier, Nicolas
%Y Rozanov, Nikolai
%Y Hashemzadeh, Maryam
%Y Arora, Akhil
%S Proceedings of the Second Workshop for REsearch on Agent Language Models (REALM 2026)
%D 2026
%8 October
%I Association for Computational Linguistics
%C Budapest, Hungary
%@ 979-8-89176-485-9
%F cao-2026-policy
%X Agent benchmarks evaluate policy compliance but assume each policy determines a unique correct action. Natural-language policies can violate this assumption through silence, ambiguity, or contradiction, admitting multiple defensible readings that a single gold trajectory cannot capture. Auditing two τ²-bench domains, we develop a taxonomy of such policy loopholes and show that affected tasks produce unreliable scores: they lower scores across different models in different ways and make every model less consistent across repeated trials. A cross-domain comparison reveals that exploitability requires both policy ambiguity and tool permissiveness: when policy complexity exceeds what tools can enforce, agents resolve gaps inconsistently and scores become unreliable. Policy specification quality sets the ceiling on evaluation quality. Benchmark developers should audit policies before collecting gold annotations.
%U https://aclanthology.org/2026.realm-1.98/
%P 1404-1419

Markdown (Informal)

[Policy Loopholes in Agent Evaluation: When Policy Ambiguity Masquerades as Agent Error](https://aclanthology.org/2026.realm-1.98/) (Cao, REALM 2026)

• Policy Loopholes in Agent Evaluation: When Policy Ambiguity Masquerades as Agent Error (Cao, REALM 2026)

ACL

• Hongliu Cao. 2026. Policy Loopholes in Agent Evaluation: When Policy Ambiguity Masquerades as Agent Error. In Proceedings of the Second Workshop for REsearch on Agent Language Models (REALM 2026), pages 1404–1419, Budapest, Hungary. Association for Computational Linguistics.