Existing methods for evaluating lexical competence in language models often rely on discriminative tasks or expensive, human-curated datasets. We introduce EvaLex, an automated framework to evaluate lexical competence in language models without gold labels. EvaLex employs a bidirectional approach: it first prompts a model to define a target word and subsequently tasks the model with retrieving that same word from its own generated definition in a separate inference session. We evaluate EvaLex on two Spanish datasets: CREA-10k-POS (7,784 words) and CREA-10k-Lemmas (4,594 lemmas), and we further validate against an expert-annotated DEA sample (100 words). We compare similarly sized LLMs (Llama, Mistral, Yi, Gemma) alongside Spanish-centric models (Gromenauer, Salamandra). EvaLex scores correlate strongly with expert manual annotations (Pearson r = 0.8539, Spearman rho = 0.8) across the four models for which manual annotations were available, supporting its use as a proxy for human judgments. Broader validation across more models and languages remains an open priority. We also observe a clear frequency effect, with Spanish-specific models remaining more robust on low-frequency terms. Overall, EvaLex provides a scalable and low-cost method for model selection and for diagnosing lexical gaps in low-resource and cross-lingual settings.

This paper describes the participation of the SINAI research group at WASSA 2022 (Empathy and Personality Detection and Emotion Classification). Specifically, we participate in Track 1 (Empathy and Distress predictions) and Track 2 (Emotion classification). We conducted extensive experiments developing different machine learning solutions in line with the state of the art in Natural Language Processing. For Track 1, a Transformer multi-output regression model is proposed. For Track 2, we aim to explore recent techniques based on Zero-Shot Learning models including a Natural Language Inference model and GPT-3, using them in an ensemble manner with a fine-tune RoBERTa model. Our team ranked 2nd in the first track and 3rd in the second track.