The rapid adoption of large language models (LLMs) in finance has enabled automated generation of in-domain texts such as earnings summaries, market analyses, and commentary on regulated disclosures. Generating accurate and accessible financial commentary from material facts poses challenges related to domain-specific language, strict factual faithfulness, and readability. Evaluating such outputs is difficult: traditional automatic metrics overlook financial correctness and adequacy, while human expert assessment is reliable but costly and subjective. This paper proposes a multidimensional evaluation protocol for generating financial commentary in Portuguese and investigates the use of LLMs as evaluators (“LLMs-as-judges”) in this high-stakes, low-resource setting. We systematically compare human expert judgments and LLM-based evaluation preferences, while also investigating complementary dimensions such as writing quality, factuality, usefulness, and simplicity. To the best of our knowledge, this is the first proposal of an evaluation protocol for this task in Portuguese. Furthermore, we analyze where LLM judgments align with or diverge from human assessments, providing practical insights and recommendations for evaluation methodologies in financial AI applications.

Este trabalho retoma bases epistemológicas e metodológicas de diversas áreas da ciência, como psicologia, linguística de corpus, sociologia e neurociência, para fundamentar princípios básicos da avaliação de sistemas baseados em Processamento de Linguagem Natural e Inteligência Artificial. Apresentamos em detalhe o estado da arte da avaliação na área, discutindo os motivos que nos levam à necessidade de automatizar a avaliação de sistemas (paradigma LLM-as-a-Judge). Introduzimos Cinco Princípios Metodológicos básicos que devem fundamentar a avaliação de sistemas baseados em IA, fundamentados tanto na teoria e experimentos destas áreas que já pesquisam a percepção humana há décadas quanto em evidências empíricas do próprio PLN.

Automatic metrics are widely used to evaluate text quality across various natural language processing tasks. Despite their convenience and scalability, the extent to which these metrics reliably reflect textual quality remains an open challenge. The LLM-as-a-judge paradigm has recently emerged, aligning more closely with human judgments by using LLMs themselves as evaluators. However, there is still a gap in such evaluations across specific domains and languages, as most prior work focuses on generic task benchmarks in English. In this paper, we examine the robustness of both traditional automatic metrics and the LLM-as-a-judge approach for assessing the quality of financial commentaries in Portuguese, an underexplored task and language that has been neglected in previous work. We introduce fine-grained perturbations into the texts generated by specialists to analyze which types of noise most significantly affect evaluation outcomes, using noise-free counterparts as references. The results highlight the weaknesses of classical metrics in this specific task and the limitations of even recent evaluation paradigms, underscoring the need to develop context- and domain-sensitive.

We present CITAR-NER-Br, a unified NER dataset for direct extraction of fine-grained legal citation subcomponents in Brazilian Portuguese. CITAR-NER-Br annotates six citation subcomponents over 270 legal documents from LeNER-Br and Ulysses Tesemõ, capturing the overlapping and discontinuous spans intrinsic to legal citations. To the best of our knowledge, this is the first unified NER dataset for the legal domain specifically designed to represent discontinuous entities, which account for 44.8% of annotations, the highest proportion among existing legal NER corpora. We establish baselines using four configurations across three LLM-based and non-LLM NER methods to support future research. Our experiments demonstrate that, while non-LLM methods achieve strong performance, they remain heavily reliant on high-quality training data. Conversely, LLM-based methods achieve overall lower performance, but require only few-shot supervision. Moreover, the traditional exact-match NER evaluation approach significantly penalizes LLM-based methods even when their outputs preserve semantic accuracy.

Legal documents frequently contain heterogeneous references to judicial precedents, making automated Precedent Reference Extraction (PRE) challenging for legal AI systems. Existing approaches rely heavily on LLM-based pipelines, which are often effective but also incur recurring prompting costs in high-volume production workloads and struggle with tracing extracted entities back to their character offsets in the source text. In light of these issues, we present Notarius, a domain-specific and production-ready hybrid framework that decomposes PRE into two stages: Overlapped NER Feature Extraction (ONFE), which extracts fine-grained precedent features, and Precedent Feature Aggregation (PFA), which reconstructs complete references for downstream applications. Our implementation integrates the Unified NER method W²NER with domain-specific heuristics. We evaluate Notarius at both feature and reference levels on a collection of judicial decisions and compare it in production against an LLM-based pipeline. Notarius achieves higher extraction quality, eliminates recurring LLM API costs for PRE, and provides span-level offset traceability. The system currently powers precedent extraction on Jusbrasil, demonstrating that discriminative, domain-specialized NER architectures remain valuable for cost-effective, high-precision legal information extraction in the LLM era.

Material facts (MF) are crucial and obligatory disclosures that can significantly influence asset values. Following their release, financial analysts embark on the meticulous and highly specialized task of crafting analyses to shed light on their impact on company assets, a challenge elevated by the daily amount of MFs released. Generative AI, with its demonstrated power of crafting coherent text, emerges as a promising solution to this task. However, while these analyses must incorporate the MF, they must also transcend it, enhancing it with vital background information, valuable and grounded recommendations, prospects, potential risks, and their underlying reasoning. In this paper, we approach this task as an instance of controllable text generation, aiming to ensure adherence to the MF and other pivotal attributes as control elements. We first explore language models’ capacity to manage this task by embedding those elements into prompts and engaging popular chatbots. A bilingual proof of concept underscores both the potential and the challenges of applying generative AI techniques to this task.

This paper describes the system submitted by our team (BabelEnconding) to SemEval-2020 Task 3: Predicting the Graded Effect of Context in Word Similarity. We propose an approach that relies on translation and multilingual language models in order to compute the contextual similarity between pairs of words. Our hypothesis is that evidence from additional languages can leverage the correlation with the human generated scores. BabelEnconding was applied to both subtasks and ranked among the top-3 in six out of eight task/language combinations and was the highest scoring system three times.