<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="ratnakar-etal-2026-latentgate">
    <titleInfo>
        <title>LatentGate: Low-Latency Semantic Routing via Frozen-Backbone Probing of Small Language Models</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Shivam</namePart>
        <namePart type="family">Ratnakar</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Abhiroop</namePart>
        <namePart type="family">Talasila</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Vinayak</namePart>
        <namePart type="given">K</namePart>
        <namePart type="family">Doifode</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-07</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (ACL 2026)</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Yunyao</namePart>
            <namePart type="family">Li</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Georg</namePart>
            <namePart type="family">Rehm</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Mei</namePart>
            <namePart type="family">Tu</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">San Diego, California, USA</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-394-4</identifier>
    </relatedItem>
    <abstract>As Multi-Agent Systems scale to hundreds of specialized agents, routing becomes a critical bottleneck. Prompt-based LLM routers deliver strong semantic reasoning but incur prohibitive latency (~1500–2000ms) and cost that scales with agent count, while embedding-based routers are fast (25–50ms) but collapse semantically similar yet functionally distinct agents. We identify *representation anisotropy*, the geometric collapse of hidden-state vectors into a narrow cone, as a key mechanism underlying embedding-based routing failure. We propose **LatentGate**, a non-generative router that extracts mean-pooled hidden states from a frozen small language model (SLM), applies PCA-whitening to resolve the anisotropy, and trains a lightweight linear probe for agent classification. Across 5 SLM backbones and 100 enterprise agents, LatentGate achieves 98.8% in-domain and 80.0% OOD accuracy on natural queries, 13–22 absolute points above embedding baselines, and 92.9% on CLINC150. It takes ~28ms to run on a T4 GPU, with the SLM forward pass independent of agent count and classification adding a negligible O(Ck) term. We demonstrate the potential of using a lightweight linear probe to enable sub-10ms warm-start retraining from user feedback, providing a foundation for continual learning in production environments. Benchmarking prompt-based routing with GPT-4.1, GPT-4.1-nano, and Gemini 2.5 Flash confirms degradation to 70–77% accuracy at 100 agents with 1500–2000ms latency, motivating non-generative alternatives.</abstract>
    <identifier type="citekey">ratnakar-etal-2026-latentgate</identifier>
    <location>
        <url>https://aclanthology.org/2026.acl-industry.153/</url>
    </location>
    <part>
        <date>2026-07</date>
        <extent unit="page">
            <start>2284</start>
            <end>2294</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T LatentGate: Low-Latency Semantic Routing via Frozen-Backbone Probing of Small Language Models
%A Ratnakar, Shivam
%A Talasila, Abhiroop
%A Doifode, Vinayak K.
%Y Li, Yunyao
%Y Rehm, Georg
%Y Tu, Mei
%S Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (ACL 2026)
%D 2026
%8 July
%I Association for Computational Linguistics
%C San Diego, California, USA
%@ 979-8-89176-394-4
%F ratnakar-etal-2026-latentgate
%X As Multi-Agent Systems scale to hundreds of specialized agents, routing becomes a critical bottleneck. Prompt-based LLM routers deliver strong semantic reasoning but incur prohibitive latency (~1500–2000ms) and cost that scales with agent count, while embedding-based routers are fast (25–50ms) but collapse semantically similar yet functionally distinct agents. We identify *representation anisotropy*, the geometric collapse of hidden-state vectors into a narrow cone, as a key mechanism underlying embedding-based routing failure. We propose **LatentGate**, a non-generative router that extracts mean-pooled hidden states from a frozen small language model (SLM), applies PCA-whitening to resolve the anisotropy, and trains a lightweight linear probe for agent classification. Across 5 SLM backbones and 100 enterprise agents, LatentGate achieves 98.8% in-domain and 80.0% OOD accuracy on natural queries, 13–22 absolute points above embedding baselines, and 92.9% on CLINC150. It takes ~28ms to run on a T4 GPU, with the SLM forward pass independent of agent count and classification adding a negligible O(Ck) term. We demonstrate the potential of using a lightweight linear probe to enable sub-10ms warm-start retraining from user feedback, providing a foundation for continual learning in production environments. Benchmarking prompt-based routing with GPT-4.1, GPT-4.1-nano, and Gemini 2.5 Flash confirms degradation to 70–77% accuracy at 100 agents with 1500–2000ms latency, motivating non-generative alternatives.
%U https://aclanthology.org/2026.acl-industry.153/
%P 2284-2294

Markdown (Informal)

[LatentGate: Low-Latency Semantic Routing via Frozen-Backbone Probing of Small Language Models](https://aclanthology.org/2026.acl-industry.153/) (Ratnakar et al., ACL 2026)

• LatentGate: Low-Latency Semantic Routing via Frozen-Backbone Probing of Small Language Models (Ratnakar et al., ACL 2026)

ACL

• Shivam Ratnakar, Abhiroop Talasila, and Vinayak K Doifode. 2026. LatentGate: Low-Latency Semantic Routing via Frozen-Backbone Probing of Small Language Models. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (ACL 2026), pages 2284–2294, San Diego, California, USA. Association for Computational Linguistics.