Valeria De Paiva

Also published as: Valeria de Paiva


pdf bib
Mathematical Entities: Corpora and Benchmarks
Jacob Collard | Valeria de Paiva | Eswaran Subrahmanian
Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)

Mathematics is a highly specialized domain with its own unique set of challenges. Despite this, there has been relatively little research on natural language processing for mathematical texts, and there are few mathematical language resources aimed at NLP. In this paper, we aim to provide annotated corpora that can be used to study the language of mathematics in different contexts, ranging from fundamental concepts found in textbooks to advanced research mathematics. We preprocess the corpora with a neural parsing model and some manual intervention to provide part-of-speech tags, lemmas, and dependency trees. In total, we provide 182397 sentences across three corpora. We then aim to test and evaluate several noteworthy natural language processing models using these corpora, to show how well they can adapt to the domain of mathematics and provide useful tools for exploring mathematical language. We evaluate several neural and symbolic models against benchmarks that we extract from the corpus metadata to show that terminology extraction and definition extraction do not easily generalize to mathematics, and that additional work is needed to achieve good performance on these metrics. Finally, we provide a learning assistant that grants access to the content of these corpora in a context-sensitive manner, utilizing text search and entity linking. Though our corpora and benchmarks provide useful metrics for evaluating mathematical language processing, further work is necessary to adapt models to mathematics in order to provide more effective learning assistants and apply NLP methods to different mathematical domains.


pdf bib
Curing the SICK and Other NLI Maladies
Aikaterini-Lida Kalouli | Hai Hu | Alexander F. Webb | Lawrence S. Moss | Valeria de Paiva
Computational Linguistics, Volume 49, Issue 1 - March 2023

Against the backdrop of the ever-improving Natural Language Inference (NLI) models, recent efforts have focused on the suitability of the current NLI datasets and on the feasibility of the NLI task as it is currently approached. Many of the recent studies have exposed the inherent human disagreements of the inference task and have proposed a shift from categorical labels to human subjective probability assessments, capturing human uncertainty. In this work, we show how neither the current task formulation nor the proposed uncertainty gradient are entirely suitable for solving the NLI challenges. Instead, we propose an ordered sense space annotation, which distinguishes between logical and common-sense inference. One end of the space captures non-sensical inferences, while the other end represents strictly logical scenarios. In the middle of the space, we find a continuum of common-sense, namely, the subjective and graded opinion of a “person on the street.” To arrive at the proposed annotation scheme, we perform a careful investigation of the SICK corpus and we create a taxonomy of annotation issues and guidelines. We re-annotate the corpus with the proposed annotation scheme, utilizing four symbolic inference systems, and then perform a thorough evaluation of the scheme by fine-tuning and testing commonly used pre-trained language models on the re-annotated SICK within various settings. We also pioneer a crowd annotation of a small portion of the MultiNLI corpus, showcasing that it is possible to adapt our scheme for annotation by non-experts on another NLI corpus. Our work shows the efficiency and benefits of the proposed mechanism and opens the way for a careful NLI task refinement.

pdf bib
Proceedings of the 4th Natural Logic Meets Machine Learning Workshop
Stergios Chatzikyriakidis | Valeria de Paiva
Proceedings of the 4th Natural Logic Meets Machine Learning Workshop


pdf bib
Extracting Mathematical Concepts from Text
Jacob Collard | Valeria de Paiva | Brendan Fong | Eswaran Subrahmanian
Proceedings of the Eighth Workshop on Noisy User-generated Text (W-NUT 2022)

We investigate different systems for extracting mathematical entities from English texts in the mathematical field of category theory as a first step for constructing a mathematical knowledge graph. We consider four different term extractors and compare their results. This small experiment showcases some of the issues with the construction and evaluation of terms extracted from noisy domain text. We also make available two open corpora in research mathematics, in particular in category theory: a small corpus of 755 abstracts from the journal TAC (3188 sentences), and a larger corpus from the nLab community wiki (15,000 sentences)


pdf bib
Hy-NLI: a Hybrid system for Natural Language Inference
Aikaterini-Lida Kalouli | Richard Crouch | Valeria de Paiva
Proceedings of the 28th International Conference on Computational Linguistics

Despite the advances in Natural Language Inference through the training of massive deep models, recent work has revealed the generalization difficulties of such models, which fail to perform on adversarial datasets with challenging linguistic phenomena. Such phenomena, however, can be handled well by symbolic systems. Thus, we propose Hy-NLI, a hybrid system that learns to identify an NLI pair as linguistically challenging or not. Based on that, it uses its symbolic or deep learning component, respectively, to make the final inference decision. We show how linguistically less complex cases are best solved by robust state-of-the-art models, like BERT and XLNet, while hard linguistic phenomena are best handled by our implemented symbolic engine. Our thorough evaluation shows that our hybrid system achieves state-of-the-art performance across mainstream and adversarial datasets and opens the way for further research into the hybrid direction.

pdf bib
XplaiNLI: Explainable Natural Language Inference through Visual Analytics
Aikaterini-Lida Kalouli | Rita Sevastjanova | Valeria de Paiva | Richard Crouch | Mennatallah El-Assady
Proceedings of the 28th International Conference on Computational Linguistics: System Demonstrations

Advances in Natural Language Inference (NLI) have helped us understand what state-of-the-art models really learn and what their generalization power is. Recent research has revealed some heuristics and biases of these models. However, to date, there is no systematic effort to capitalize on those insights through a system that uses these to explain the NLI decisions. To this end, we propose XplaiNLI, an eXplainable, interactive, visualization interface that computes NLI with different methods and provides explanations for the decisions made by the different approaches.


pdf bib
Proceedings of the Sixth Workshop on Natural Language and Computer Science
Robin Cooper | Valeria de Paiva | Lawrence S. Moss
Proceedings of the Sixth Workshop on Natural Language and Computer Science

pdf bib
GKR: Bridging the Gap between Symbolic/structural and Distributional Meaning Representations
Aikaterini-Lida Kalouli | Richard Crouch | Valeria de Paiva
Proceedings of the First International Workshop on Designing Meaning Representations

Three broad approaches have been attempted to combine distributional and structural/symbolic aspects to construct meaning representations: a) injecting linguistic features into distributional representations, b) injecting distributional features into symbolic representations or c) combining structural and distributional features in the final representation. This work focuses on an example of the third and less studied approach: it extends the Graphical Knowledge Representation (GKR) to include distributional features and proposes a division of semantic labour between the distributional and structural/symbolic features. We propose two extensions of GKR that clearly show this division and empirically test one of the proposals on an NLI dataset with hard compositional pairs.

pdf bib
Explaining Simple Natural Language Inference
Aikaterini-Lida Kalouli | Annebeth Buis | Livy Real | Martha Palmer | Valeria de Paiva
Proceedings of the 13th Linguistic Annotation Workshop

The vast amount of research introducing new corpora and techniques for semi-automatically annotating corpora shows the important role that datasets play in today’s research, especially in the machine learning community. This rapid development raises concerns about the quality of the datasets created and consequently of the models trained, as recently discussed with respect to the Natural Language Inference (NLI) task. In this work we conduct an annotation experiment based on a small subset of the SICK corpus. The experiment reveals several problems in the annotation guidelines, and various challenges of the NLI task itself. Our quantitative evaluation of the experiment allows us to assign our empirical observations to specific linguistic phenomena and leads us to recommendations for future annotation tasks, for NLI and possibly for other tasks.

pdf bib
Composing Noun Phrase Vector Representations
Aikaterini-Lida Kalouli | Valeria de Paiva | Richard Crouch
Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019)

Vector representations of words have seen an increasing success over the past years in a variety of NLP tasks. While there seems to be a consensus about the usefulness of word embeddings and how to learn them, it is still unclear which representations can capture the meaning of phrases or even whole sentences. Recent work has shown that simple operations outperform more complex deep architectures. In this work, we propose two novel constraints for computing noun phrase vector representations. First, we propose that the semantic and not the syntactic contribution of each component of a noun phrase should be considered, so that the resulting composed vectors express more of the phrase meaning. Second, the composition process of the two phrase vectors should apply suitable dimensions’ selection in a way that specific semantic features captured by the phrase’s meaning become more salient. Our proposed methods are compared to 11 other approaches, including popular baselines and a neural net architecture, and are evaluated across 6 tasks and 2 datasets. Our results show that these constraints lead to more expressive phrase representations and can be applied to other state-of-the-art methods to improve their performance.

pdf bib
Portuguese Manners of Speaking
Valeria de Paiva | Alexandre Rademaker
Proceedings of the 10th Global Wordnet Conference

Lexical resources need to be as complete as possible. Very little work seems to have been done on adverbs, the smallest part of speech class in Princeton WordNet counting the number of synsets. Amongst adverbs, manner adverbs ending in ‘-ly’ seem the easiest to work with, as their meaning is almost the same as the one of the associated adjective. This phenomenon seems to be parallel in English and Portuguese, where these manner adverbs finish in the suffix ‘-mente’. We use this correspondence to improve the coverage of adverbs in the lexical resource OpenWordNet-PT, a wordnet for Portuguese.


pdf bib
Extending Wordnet to Geological Times
Henrique Muniz | Fabricio Chalub | Alexandre Rademaker | Valeria De Paiva
Proceedings of the 9th Global Wordnet Conference

This paper describes work extending Princeton WordNet to the domain of geological texts, associated with the time periods of the geological eras of the Earth History. We intend this extension to be considered as an example for any other domain extension that we might want to pursue. To provide this extension, we first produce a textual version of Princeton WordNet. Then we map a fragment of the International Commission on Stratigraphy (ICS) ontologies to WordNet and create the appropriate new synsets. We check the extended ontology on a small corpus of sentences from Gas and Oil technical reports and realize that more work needs to be done, as we need new words, new senses and new compounds in our extended WordNet.


pdf bib
CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies
Daniel Zeman | Martin Popel | Milan Straka | Jan Hajič | Joakim Nivre | Filip Ginter | Juhani Luotolahti | Sampo Pyysalo | Slav Petrov | Martin Potthast | Francis Tyers | Elena Badmaeva | Memduh Gokirmak | Anna Nedoluzhko | Silvie Cinková | Jan Hajič jr. | Jaroslava Hlaváčová | Václava Kettnerová | Zdeňka Urešová | Jenna Kanerva | Stina Ojala | Anna Missilä | Christopher D. Manning | Sebastian Schuster | Siva Reddy | Dima Taji | Nizar Habash | Herman Leung | Marie-Catherine de Marneffe | Manuela Sanguinetti | Maria Simi | Hiroshi Kanayama | Valeria de Paiva | Kira Droganova | Héctor Martínez Alonso | Çağrı Çöltekin | Umut Sulubacak | Hans Uszkoreit | Vivien Macketanz | Aljoscha Burchardt | Kim Harris | Katrin Marheinecke | Georg Rehm | Tolga Kayadelen | Mohammed Attia | Ali Elkahky | Zhuoran Yu | Emily Pitler | Saran Lertpradit | Michael Mandl | Jesse Kirchner | Hector Fernandez Alcalde | Jana Strnadová | Esha Banerjee | Ruli Manurung | Antonio Stella | Atsuko Shimada | Sookyoung Kwak | Gustavo Mendonça | Tatiana Lando | Rattima Nitisaroj | Josie Li
Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies

The Conference on Computational Natural Language Learning (CoNLL) features a shared task, in which participants train and test their learning systems on the same data sets. In 2017, the task was devoted to learning dependency parsers for a large number of languages, in a real-world setting without any gold-standard annotation on input. All test sets followed a unified annotation scheme, namely that of Universal Dependencies. In this paper, we define the task and evaluation methodology, describe how the data sets were prepared, report and analyze the main results, and provide a brief categorization of the different approaches of the participating systems.

pdf bib
Universal Dependencies for Portuguese
Alexandre Rademaker | Fabricio Chalub | Livy Real | Cláudia Freitas | Eckhard Bick | Valeria de Paiva
Proceedings of the Fourth International Conference on Dependency Linguistics (Depling 2017)

pdf bib
Textual Inference: getting logic from humans
Aikaterini-Lida Kalouli | Livy Real | Valeria de Paiva
Proceedings of the 12th International Conference on Computational Semantics (IWCS) — Short papers

pdf bib
Correcting Contradictions
Aikaterini-Lida Kalouli | Valeria de Paiva | Livy Real
Proceedings of the Computing Natural Language Inference Workshop


pdf bib
Semantic Links for Portuguese
Fabricio Chalub | Livy Real | Alexandre Rademaker | Valeria de Paiva
Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16)

This paper describes work on incorporating Princenton’s WordNet morphosemantics links to the fabric of the Portuguese OpenWordNet-PT. Morphosemantic links are relations between verbs and derivationally related nouns that are semantically typed (such as for tune-tuner ― in Portuguese “afinar-afinador” – linked through an “agent” link). Morphosemantic links have been discussed for Princeton’s WordNet for a while, but have not been added to the official database. These links are very useful, they help us to improve our Portuguese WordNet. Thus we discuss the integration of these links in our base and the issues we encountered with the integration.

pdf bib
An overview of Portuguese WordNets
Valeria de Paiva | Livy Real | Hugo Gonçalo Oliveira | Alexandre Rademaker | Cláudia Freitas | Alberto Simões
Proceedings of the 8th Global WordNet Conference (GWC)

Semantic relations between words are key to building systems that aim to understand and manipulate language. For English, the “de facto” standard for representing this kind of knowledge is Princeton’s WordNet. Here, we describe the wordnet-like resources currently available for Portuguese: their origins, methods of creation, sizes, and usage restrictions. We start tackling the problem of comparing them, but only in quantitative terms. Finally, we sketch ideas for potential collaboration between some of the projects that produce Portuguese wordnets.


pdf bib
Seeing is Correcting: curating lexical resources using social interfaces
Livy Real | Fabricio Chalub | Valeria de Paiva | Claudia Freitas | Alexandre Rademaker
Proceedings of the 4th Workshop on Linked Data in Linguistics: Resources and Applications


pdf bib
Embedding NomLex-BR nominalizations into OpenWordnet-PT
Alexandre Rademaker | Valeria de Paiva | Gerard de Melo | Livy Maria Real Coelho
Proceedings of the Seventh Global Wordnet Conference

pdf bib
OpenWordNet-PT: A Project Report
Alexandre Rademaker | Valeria de Paiva | Gerard de Melo | Livy Real | Maira Gatti
Proceedings of the Seventh Global Wordnet Conference

pdf bib
Annie Zaenen | Cleo Condoravdi | Valeria de Paiva
Linguistic Issues in Language Technology, Volume 9, 2014 - Perspectives on Semantic Representations for Textual Inference

pdf bib
NomLex-PT: A Lexicon of Portuguese Nominalizations
Valeria de Paiva | Livy Real | Alexandre Rademaker | Gerard de Melo
Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14)

This paper presents NomLex-PT, a lexical resource describing Portuguese nominalizations. NomLex-PT connects verbs to their nominalizations, thereby enabling NLP systems to observe the potential semantic relationships between the two words when analysing a text. NomLex-PT is freely available and encoded in RDF for easy integration with other resources. Most notably, we have integrated NomLex-PT with OpenWordNet-PT, an open Portuguese Wordnet.


pdf bib
Where’s the meeting that was cancelled? existential implications of transitive verbs
Patricia Amaral | Valeria de Paiva | Cleo Condoravdi | Annie Zaenen
Proceedings of the 3rd Workshop on Cognitive Aspects of the Lexicon

pdf bib
OpenWordNet-PT: An Open Brazilian Wordnet for Reasoning
Valeria de Paiva | Alexandre Rademaker | Gerard de Melo
Proceedings of COLING 2012: Demonstration Papers


pdf bib
Context Inducing Nouns
Charlotte Price | Valeria de Paiva | Tracy Holloway King
Coling 2008: Proceedings of the workshop on Knowledge and Reasoning for Answering Questions

pdf bib
Designing Testsuites for Grammar-based Systems in Applications
Valeria de Paiva | Tracy Holloway King
Coling 2008: Proceedings of the workshop on Grammar Engineering Across Frameworks


pdf bib
Precision-focused Textual Inference
Daniel Bobrow | Dick Crouch | Tracy Holloway King | Cleo Condoravdi | Lauri Karttunen | Rowan Nairn | Valeria de Paiva | Annie Zaenen
Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing


pdf bib
Entailment, intensionality and text understanding
Cleo Condoravdi | Dick Crouch | Valeria de Paiva | Reinhard Stolle | Daniel G. Bobrow
Proceedings of the HLT-NAACL 2003 Workshop on Text Meaning