Exploration of Sentence Representations in Spanish BERT-like Models

Gonzalo Herrera; Aiala Rosá; Luis Chiruzzo

doi:10.63317/5moein87oxiw

Exploration of Sentence Representations in Spanish BERT-like Models

Gonzalo Herrera, Aiala Rosá, Luis Chiruzzo

Correct Metadata for

Use this form to create a GitHub issue with structured data describing the correction. You will need a GitHub account. Once you create that issue, the correction will be reviewed by a staff member.

⚠️ Mobile Users: Submitting this form to create a new issue will only work with github.com, not the GitHub Mobile app.

Important: The Anthology treat PDFs as authoritative. Please use this form only to correct data that is out of line with the PDF. See our corrections guidelines if you need to change the PDF.

Title Adjust the title. Retain tags such as <fixed-case>.

Authors Adjust author names and order to match the PDF.

Abstract Correct abstract if needed. Retain XML formatting tags such as <tex-math>. You may use ... for bold, ... for italic, ... for underline, <sc>...</sc> for small-caps, <tt>...<tt> for typewriter text, <url>...</url> for URLs, <a href=...> for hyperlinks, and <par/> for paragraph breaks.

Verification against PDF Ensure that the new title/authors match the snapshot below. (If there is no snapshot or it is too small, consult the external publication .)

Authors concatenated from the text boxes above:

ALL author names match the snapshot above—including middle initials, hyphens, and accents.

Abstract

Transformer-based language models, ubiquitous in NLP nowadays, generate internal representations (embeddings) of words and sentences. Yet, systematic comparisons of embedding strategies from various models remain limited. In this work, we evaluate Spanish embeddings from several BERT-like models (BETO, multilingual BERT, XLM-RoBERTa, ROUBERTa) to understand their syntactic and semantic capabilities across layers. We propose novel sentence-level analogy tests to probe generalization. Results show tasks like verb negation or word reordering perform best with embeddings from earlier layers, while nuanced semantic distinctions—such as agent or patient gender—are better captured by deeper layers. Our findings provide guidelines for embedding strategies and offer a foundation for further NLP research.

Anthology ID:: 2026.lanlp-1.8
Volume:: Proceedings of LANLP: Bridging Ibero and Latin American NLP Communities
Month:: May
Year:: 2026
Address:: Palma, Mallorca (Spain)
Editors:: German Rigau Claramunt, Pablo Gamallo, Rafael Muñoz Guillena, Luis Chiruzzo, Eugenio Martínez Cámara
Venues:: LANLP | WS
SIG:
Publisher:: ELRA Language Resources Association (ELRA)
Note:
Pages:: 55–65
Language:
External URL:: https://lrec.elra.info/lrec2026-ws-lanlp-08
DOI:: 10.63317/5moein87oxiw
Bibkey:
Cite (ACL):: Gonzalo Herrera, Aiala Rosá, and Luis Chiruzzo. 2026. Exploration of Sentence Representations in Spanish BERT-like Models. In Proceedings of LANLP: Bridging Ibero and Latin American NLP Communities, pages 55–65, Palma, Mallorca (Spain). ELRA Language Resources Association (ELRA).
Cite (Informal):: Exploration of Sentence Representations in Spanish BERT-like Models (Herrera et al., LANLP 2026)
Copy Citation:

External Cite Search Fix data

Export citation

BibTeX
MODS XML
Endnote
Preformatted

@inproceedings{herrera-etal-2026-exploration,
    title = "Exploration of Sentence Representations in {S}panish {BERT}-like Models",
    author = "Herrera, Gonzalo  and
      Ros{\'a}, Aiala  and
      Chiruzzo, Luis",
    editor = "Claramunt, German Rigau  and
      Gamallo, Pablo  and
      Mu{\~n}oz Guillena, Rafael  and
      Chiruzzo, Luis  and
      Mart{\'i}nez C{\'a}mara, Eugenio",
    booktitle = "Proceedings of {LANLP}: Bridging {I}bero and {L}atin {A}merican {NLP} Communities",
    month = may,
    year = "2026",
    address = "Palma, Mallorca (Spain)",
    publisher = "ELRA Language Resources Association (ELRA)",
    url = "https://aclanthology.org/2026.lanlp-1.8/",
    doi = "10.63317/5moein87oxiw",
    pages = "55--65",
    abstract = "Transformer-based language models, ubiquitous in NLP nowadays, generate internal representations (embeddings) of words and sentences. Yet, systematic comparisons of embedding strategies from various models remain limited. In this work, we evaluate Spanish embeddings from several BERT-like models (BETO, multilingual BERT, XLM-RoBERTa, ROUBERTa) to understand their syntactic and semantic capabilities across layers. We propose novel sentence-level analogy tests to probe generalization. Results show tasks like verb negation or word reordering perform best with embeddings from earlier layers, while nuanced semantic distinctions{---}such as agent or patient gender{---}are better captured by deeper layers. Our findings provide guidelines for embedding strategies and offer a foundation for further NLP research."
}

Download as File

<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="herrera-etal-2026-exploration">
    <titleInfo>
        <title>Exploration of Sentence Representations in Spanish BERT-like Models</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Gonzalo</namePart>
        <namePart type="family">Herrera</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Aiala</namePart>
        <namePart type="family">Rosá</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Luis</namePart>
        <namePart type="family">Chiruzzo</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-05</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of LANLP: Bridging Ibero and Latin American NLP Communities</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">German</namePart>
            <namePart type="given">Rigau</namePart>
            <namePart type="family">Claramunt</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Pablo</namePart>
            <namePart type="family">Gamallo</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Rafael</namePart>
            <namePart type="family">Muñoz Guillena</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Luis</namePart>
            <namePart type="family">Chiruzzo</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Eugenio</namePart>
            <namePart type="family">Martínez Cámara</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>ELRA Language Resources Association (ELRA)</publisher>
            <place>
                <placeTerm type="text">Palma, Mallorca (Spain)</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Transformer-based language models, ubiquitous in NLP nowadays, generate internal representations (embeddings) of words and sentences. Yet, systematic comparisons of embedding strategies from various models remain limited. In this work, we evaluate Spanish embeddings from several BERT-like models (BETO, multilingual BERT, XLM-RoBERTa, ROUBERTa) to understand their syntactic and semantic capabilities across layers. We propose novel sentence-level analogy tests to probe generalization. Results show tasks like verb negation or word reordering perform best with embeddings from earlier layers, while nuanced semantic distinctions—such as agent or patient gender—are better captured by deeper layers. Our findings provide guidelines for embedding strategies and offer a foundation for further NLP research.</abstract>
    <identifier type="citekey">herrera-etal-2026-exploration</identifier>
    <identifier type="doi">10.63317/5moein87oxiw</identifier>
    <location>
        <url>https://aclanthology.org/2026.lanlp-1.8/</url>
    </location>
    <part>
        <date>2026-05</date>
        <extent unit="page">
            <start>55</start>
            <end>65</end>
        </extent>
    </part>
</mods>
</modsCollection>

Download as File

%0 Conference Proceedings
%T Exploration of Sentence Representations in Spanish BERT-like Models
%A Herrera, Gonzalo
%A Rosá, Aiala
%A Chiruzzo, Luis
%Y Claramunt, German Rigau
%Y Gamallo, Pablo
%Y Muñoz Guillena, Rafael
%Y Chiruzzo, Luis
%Y Martínez Cámara, Eugenio
%S Proceedings of LANLP: Bridging Ibero and Latin American NLP Communities
%D 2026
%8 May
%I ELRA Language Resources Association (ELRA)
%C Palma, Mallorca (Spain)
%F herrera-etal-2026-exploration
%X Transformer-based language models, ubiquitous in NLP nowadays, generate internal representations (embeddings) of words and sentences. Yet, systematic comparisons of embedding strategies from various models remain limited. In this work, we evaluate Spanish embeddings from several BERT-like models (BETO, multilingual BERT, XLM-RoBERTa, ROUBERTa) to understand their syntactic and semantic capabilities across layers. We propose novel sentence-level analogy tests to probe generalization. Results show tasks like verb negation or word reordering perform best with embeddings from earlier layers, while nuanced semantic distinctions—such as agent or patient gender—are better captured by deeper layers. Our findings provide guidelines for embedding strategies and offer a foundation for further NLP research.
%R 10.63317/5moein87oxiw
%U https://aclanthology.org/2026.lanlp-1.8/
%U https://doi.org/10.63317/5moein87oxiw
%P 55-65

Download as File

Markdown (Informal)

[Exploration of Sentence Representations in Spanish BERT-like Models](https://aclanthology.org/2026.lanlp-1.8/) (Herrera et al., LANLP 2026)

Exploration of Sentence Representations in Spanish BERT-like Models (Herrera et al., LANLP 2026)

ACL

Gonzalo Herrera, Aiala Rosá, and Luis Chiruzzo. 2026. Exploration of Sentence Representations in Spanish BERT-like Models. In Proceedings of LANLP: Bridging Ibero and Latin American NLP Communities, pages 55–65, Palma, Mallorca (Spain). ELRA Language Resources Association (ELRA).