Matthew Purver


2021

pdf bib
Rare-Class Dialogue Act Tagging for Alzheimer’s Disease Diagnosis
Shamila Nasreen | Julian Hough | Matthew Purver
Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue

Alzheimer’s Disease (AD) is associated with many characteristic changes, not only in an individual’s language but also in the interactive patterns observed in dialogue. The most indicative changes of this latter kind tend to be associated with relatively rare dialogue acts (DAs), such as those involved in clarification exchanges and responses to particular kinds of questions. However, most existing work in DA tagging focuses on improving average performance, effectively prioritizing more frequent classes; it thus gives a poor performance on these rarer classes and is not suited for application to AD analysis. In this paper, we investigate tagging specifically for rare class DAs, using a hierarchical BiLSTM model with various ways of incorporating information from previous utterances and DA tags in context. We show that this can give good performance for rare DA classes on both the general Switchboard corpus (SwDA) and an AD-specific conversational dataset, the Carolinas Conversation Collection (CCC); and that the tagger outputs then contribute useful information for distinguishing patients with and without AD

pdf bib
Mitigating Topic Bias when Detecting Decisions in Dialogue
Mladen Karan | Prashant Khare | Patrick Healey | Matthew Purver
Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue

This work revisits the task of detecting decision-related utterances in multi-party dialogue. We explore performance of a traditional approach and a deep learning-based approach based on transformer language models, with the latter providing modest improvements. We then analyze topic bias in the models using topic information obtained by manual annotation. Our finding is that when detecting some types of decisions in our data, models rely more on topic specific words that decisions are about rather than on words that more generally indicate decision making. We further explore this by removing topic information from the train data. We show that this resolves the bias issues to an extent and, surprisingly, sometimes even boosts performance.

pdf bib
Zero-shot Cross-lingual Content Filtering: Offensive Language and Hate Speech Detection
Andraž Pelicon | Ravi Shekhar | Matej Martinc | Blaž Škrlj | Matthew Purver | Senja Pollak
Proceedings of the EACL Hackashop on News Media Content Analysis and Automated Report Generation

We present a system for zero-shot cross-lingual offensive language and hate speech classification. The system was trained on English datasets and tested on a task of detecting hate speech and offensive social media content in a number of languages without any additional training. Experiments show an impressive ability of both models to generalize from English to other languages. There is however an expected gap in performance between the tested cross-lingual models and the monolingual models. The best performing model (offensive content classifier) is available online as a REST API.

pdf bib
EMBEDDIA Tools, Datasets and Challenges: Resources and Hackathon Contributions
Senja Pollak | Marko Robnik-Šikonja | Matthew Purver | Michele Boggia | Ravi Shekhar | Marko Pranjić | Salla Salmela | Ivar Krustok | Tarmo Paju | Carl-Gustav Linden | Leo Leppänen | Elaine Zosa | Matej Ulčar | Linda Freienthal | Silver Traat | Luis Adrián Cabrera-Diego | Matej Martinc | Nada Lavrač | Blaž Škrlj | Martin Žnidaršič | Andraž Pelicon | Boshko Koloski | Vid Podpečan | Janez Kranjc | Shane Sheehan | Emanuela Boros | Jose G. Moreno | Antoine Doucet | Hannu Toivonen
Proceedings of the EACL Hackashop on News Media Content Analysis and Automated Report Generation

This paper presents tools and data sources collected and released by the EMBEDDIA project, supported by the European Union’s Horizon 2020 research and innovation program. The collected resources were offered to participants of a hackathon organized as part of the EACL Hackashop on News Media Content Analysis and Automated Report Generation in February 2021. The hackathon had six participating teams who addressed different challenges, either from the list of proposed challenges or their own news-industry-related tasks. This paper goes beyond the scope of the hackathon, as it brings together in a coherent and compact form most of the resources developed, collected and released by the EMBEDDIA project. Moreover, it constitutes a handy source for news media industry and researchers in the fields of Natural Language Processing and Social Science.

pdf bib
Communicative Grounding of Analogical Explanations in Dialogue: A Corpus Study of Conversational Management Acts and Statistical Sequence Models for Tutoring through Analogy
Jorge Del-Bosque-Trevino | Julian Hough | Matthew Purver
Proceedings of the Reasoning and Interaction Conference (ReInAct 2021)

We present a conversational management act (CMA) annotation schema for one-to-one tutorial dialogue sessions where a tutor uses an analogy to teach a student a concept. CMAs are more fine-grained sub-utterance acts compared to traditional dialogue act mark-up. The schema achieves an inter-annotator agreement (IAA) Cohen Kappa score of at least 0.66 across all 10 classes. We annotate a corpus of analogical episodes with the schema and develop statistical sequence models from the corpus which predict tutor content related decisions, in terms of the selection of the analogical component (AC) and tutor conversational management act (TCMA) to deploy at the current utterance, given the student’s behaviour. CRF sequence classifiers perform well on AC selection and robustly on TCMA selection, achieving respective accuracies of 61.9% and 56.3% on a cross-validation experiment over the corpus.

pdf bib
Towards Robustness of Text-to-SQL Models against Synonym Substitution
Yujian Gan | Xinyun Chen | Qiuping Huang | Matthew Purver | John R. Woodward | Jinxia Xie | Pengsheng Huang
Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)

Recently, there has been significant progress in studying neural networks to translate text descriptions into SQL queries. Despite achieving good performance on some public benchmarks, existing text-to-SQL models typically rely on the lexical matching between words in natural language (NL) questions and tokens in table schemas, which may render the models vulnerable to attacks that break the schema linking mechanism. In this work, we investigate the robustness of text-to-SQL models to synonym substitution. In particular, we introduce Spider-Syn, a human-curated dataset based on the Spider benchmark for text-to-SQL translation. NL questions in Spider-Syn are modified from Spider, by replacing their schema-related words with manually selected synonyms that reflect real-world question paraphrases. We observe that the accuracy dramatically drops by eliminating such explicit correspondence between NL questions and table schemas, even if the synonyms are not adversarially selected to conduct worst-case attacks. Finally, we present two categories of approaches to improve the model robustness. The first category of approaches utilizes additional synonym annotations for table schemas by modifying the model input, while the second category is based on adversarial training. We demonstrate that both categories of approaches significantly outperform their counterparts without the defense, and the first category of approaches are more effective.

2020

pdf bib
CoSimLex: A Resource for Evaluating Graded Word Similarity in Context
Carlos Santos Armendariz | Matthew Purver | Matej Ulčar | Senja Pollak | Nikola Ljubešić | Mark Granroth-Wilding
Proceedings of the 12th Language Resources and Evaluation Conference

State of the art natural language processing tools are built on context-dependent word embeddings, but no direct method for evaluating these representations currently exists. Standard tasks and datasets for intrinsic evaluation of embeddings are based on judgements of similarity, but ignore context; standard tasks for word sense disambiguation take account of context but do not provide continuous measures of meaning similarity. This paper describes an effort to build a new dataset, CoSimLex, intended to fill this gap. Building on the standard pairwise similarity task of SimLex-999, it provides context-dependent similarity measures; covers not only discrete differences in word sense but more subtle, graded changes in meaning; and covers not only a well-resourced language (English) but a number of less-resourced languages. We define the task and evaluation metrics, outline the dataset collection methodology, and describe the status of the dataset so far.

pdf bib
How Furiously Can Colorless Green Ideas Sleep? Sentence Acceptability in Context
Jey Han Lau | Carlos Armendariz | Shalom Lappin | Matthew Purver | Chang Shu
Transactions of the Association for Computational Linguistics, Volume 8

We study the influence of context on sentence acceptability. First we compare the acceptability ratings of sentences judged in isolation, with a relevant context, and with an irrelevant context. Our results show that context induces a cognitive load for humans, which compresses the distribution of ratings. Moreover, in relevant contexts we observe a discourse coherence effect that uniformly raises acceptability. Next, we test unidirectional and bidirectional language models in their ability to predict acceptability ratings. The bidirectional models show very promising results, with the best model achieving a new state-of-the-art for unsupervised acceptability prediction. The two sets of experiments provide insights into the cognitive aspects of sentence processing and central issues in the computational modeling of text and discourse.

pdf bib
SemEval-2020 Task 3: Graded Word Similarity in Context
Carlos Santos Armendariz | Matthew Purver | Senja Pollak | Nikola Ljubešić | Matej Ulčar | Ivan Vulić | Mohammad Taher Pilehvar
Proceedings of the Fourteenth Workshop on Semantic Evaluation

This paper presents the Graded Word Similarity in Context (GWSC) task which asked participants to predict the effects of context on human perception of similarity in English, Croatian, Slovene and Finnish. We received 15 submissions and 11 system description papers. A new dataset (CoSimLex) was created for evaluation in this task: it contains pairs of words, each annotated within two different contexts. Systems beat the baselines by significant margins, but few did well in more than one language or subtask. Almost every system employed a Transformer model, but with many variations in the details: WordNet sense embeddings, translation of contexts, TF-IDF weightings, and the automatic creation of datasets for fine-tuning were all used to good effect.

pdf bib
Temporal Mental Health Dynamics on Social Media
Tom Tabak | Matthew Purver
Proceedings of the 1st Workshop on NLP for COVID-19 (Part 2) at EMNLP 2020

We describe a set of experiments for building a temporal mental health dynamics system. We utilise a pre-existing methodology for distant- supervision of mental health data mining from social media platforms and deploy the system during the global COVID-19 pandemic as a case study. Despite the challenging nature of the task, we produce encouraging results, both explicit to the global pandemic and implicit to a global phenomenon, Christmas Depres- sion, supported by the literature. We propose a methodology for providing insight into tem- poral mental health dynamics to be utilised for strategic decision-making.

pdf bib
A Review of Cross-Domain Text-to-SQL Models
Yujian Gan | Matthew Purver | John R. Woodward
Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing: Student Research Workshop

WikiSQL and Spider, the large-scale cross-domain text-to-SQL datasets, have attracted much attention from the research community. The leaderboards of WikiSQL and Spider show that many researchers propose their models trying to solve the text-to-SQL problem. This paper first divides the top models in these two leaderboards into two paradigms. We then present details not mentioned in their original paper by evaluating the key components, including schema linking, pretrained word embeddings, and reasoning assistance modules. Based on the analysis of these models, we want to promote understanding of the text-to-SQL field and find out some interesting future works, for example, it is worth studying the text-to-SQL problem in an environment where it is more challenging to build schema linking and also worth studying combing the advantage of each model toward text-to-SQL.

2019

pdf bib
Proceedings of the IWCS Workshop Vector Semantics for Discourse and Dialogue
Mehrnoosh Sadrzadeh | Matthew Purver | Arash Eshghi | Julian Hough | Ruth Kempson | Patrick G. T. Healey
Proceedings of the IWCS Workshop Vector Semantics for Discourse and Dialogue

2017

pdf bib
Incongruent Headlines: Yet Another Way to Mislead Your Readers
Sophie Chesney | Maria Liakata | Massimo Poesio | Matthew Purver
Proceedings of the 2017 EMNLP Workshop: Natural Language Processing meets Journalism

This paper discusses the problem of incongruent headlines: those which do not accurately represent the information contained in the article with which they occur. We emphasise that this phenomenon should be considered separately from recognised problematic headline types such as clickbait and sensationalism, arguing that existing natural language processing (NLP) methods applied to these related concepts are not appropriate for the automatic detection of headline incongruence, as an analysis beyond stylistic traits is necessary. We therefore suggest a number of alternative methodologies that may be appropriate to the task at hand as a foundation for future work in this area. In addition, we provide an analysis of existing data sets which are related to this work, and motivate the need for a novel data set in this domain.

pdf bib
A Geometric Method for Detecting Semantic Coercion
Stephen McGregor | Elisabetta Jezek | Matthew Purver | Geraint Wiggins
IWCS 2017 - 12th International Conference on Computational Semantics - Long papers

2016

pdf bib
Robust Co-occurrence Quantification for Lexical Distributional Semantics
Dmitrijs Milajevs | Mehrnoosh Sadrzadeh | Matthew Purver
Proceedings of the ACL 2016 Student Research Workshop

pdf bib
Proceedings of the INLG 2016 Workshop on Computational Creativity in Natural Language Generation
Matthew Purver | Pablo Gervás | Sascha Griffiths
Proceedings of the INLG 2016 Workshop on Computational Creativity in Natural Language Generation

pdf bib
Process Based Evaluation of Computer Generated Poetry
Stephen McGregor | Matthew Purver | Geraint Wiggins
Proceedings of the INLG 2016 Workshop on Computational Creativity in Natural Language Generation

2015

pdf bib
Proceedings of the 11th International Conference on Computational Semantics
Matthew Purver | Mehrnoosh Sadrzadeh | Matthew Stone
Proceedings of the 11th International Conference on Computational Semantics

pdf bib
Feedback in Conversation as Incremental Semantic Update
Arash Eshghi | Christine Howes | Eleni Gregoromichelaki | Julian Hough | Matthew Purver
Proceedings of the 11th International Conference on Computational Semantics

pdf bib
Proceedings of the 15th European Workshop on Natural Language Generation (ENLG)
Anya Belz | Albert Gatt | François Portet | Matthew Purver
Proceedings of the 15th European Workshop on Natural Language Generation (ENLG)

2014

pdf bib
Probabilistic Type Theory for Incremental Dialogue Processing
Julian Hough | Matthew Purver
Proceedings of the EACL 2014 Workshop on Type Theory and Natural Language Semantics (TTNLS)

pdf bib
Investigating the Contribution of Distributional Semantic Information for Dialogue Act Classification
Dmitrijs Milajevs | Matthew Purver
Proceedings of the 2nd Workshop on Continuous Vector Space Models and their Compositionality (CVSC)

pdf bib
Linguistic Indicators of Severity and Progress in Online Text-based Therapy for Depression
Christine Howes | Matthew Purver | Rose McCabe
Proceedings of the Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality

pdf bib
A Simple Baseline for Discriminating Similar Languages
Matthew Purver
Proceedings of the First Workshop on Applying NLP Tools to Similar Languages, Varieties and Dialects

pdf bib
Strongly Incremental Repair Detection
Julian Hough | Matthew Purver
Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)

pdf bib
Evaluating Neural Word Representations in Tensor-Based Compositional Settings
Dmitrijs Milajevs | Dimitri Kartsaklis | Mehrnoosh Sadrzadeh | Matthew Purver
Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)

2013

pdf bib
Probabilistic induction for an incremental semantic grammar
Arash Eshghi | Matthew Purver | Julian Hough
Proceedings of the 10th International Conference on Computational Semantics (IWCS 2013) – Long Papers

pdf bib
Investigating Topic Modelling for Therapy Dialogue Analysis
Christine Howes | Matthew Purver | Rose McCabe
Proceedings of the IWCS 2013 Workshop on Computational Semantics in Clinical Text (CSCT 2013)

pdf bib
Incremental Grammar Induction from Child-Directed Dialogue Utterances
Arash Eshghi | Julian Hough | Matthew Purver
Proceedings of the Fourth Annual Workshop on Cognitive Modeling and Computational Linguistics (CMCL)

2012

pdf bib
Predicting Adherence to Treatment for Schizophrenia from Dialogue Transcripts
Christine Howes | Matthew Purver | Rose McCabe | Patrick G. T. Healey | Mary Lavelle
Proceedings of the 13th Annual Meeting of the Special Interest Group on Discourse and Dialogue

pdf bib
Experimenting with Distant Supervision for Emotion Classification
Matthew Purver | Stuart Battersby
Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics

2011

pdf bib
Incremental Semantic Construction in a Dialogue System
Matthew Purver | Arash Eshghi | Julian Hough
Proceedings of the Ninth International Conference on Computational Semantics (IWCS 2011)

2009

pdf bib
Proceedings of the SIGDIAL 2009 Conference
Patrick Healey | Roberto Pieraccini | Donna Byron | Steve Young | Matthew Purver
Proceedings of the SIGDIAL 2009 Conference

pdf bib
Split Utterances in Dialogue: a Corpus Study
Matthew Purver | Christine Howes | Eleni Gregoromichelaki | Patrick Healey
Proceedings of the SIGDIAL 2009 Conference

pdf bib
Cascaded Lexicalised Classifiers for Second-Person Reference Resolution
Matthew Purver | Raquel Fernández | Matthew Frampton | Stanley Peters
Proceedings of the SIGDIAL 2009 Conference

2008

pdf bib
Modelling and Detecting Decisions in Multi-party Dialogue
Raquel Fernández | Matthew Frampton | Patrick Ehlen | Matthew Purver | Stanley Peters
Proceedings of the 9th SIGdial Workshop on Discourse and Dialogue

2007

pdf bib
A Conversational In-Car Dialog System
Baoshi Yan | Fuliang Weng | Zhe Feng | Florin Ratiu | Madhuri Raya | Yao Meng | Sebastian Varges | Matthew Purver | Annie Lien | Tobias Scheideck | Badri Raghunathan | Feng Lin | Rohit Mishra | Brian Lathrop | Zhaoxia Zhang | Harry Bratt | Stanley Peters
Proceedings of Human Language Technologies: The Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT)

pdf bib
Detecting and Summarizing Action Items in Multi-Party Dialogue
Matthew Purver | John Dowding | John Niekrasz | Patrick Ehlen | Sharareh Noorbaloochi | Stanley Peters
Proceedings of the 8th SIGdial Workshop on Discourse and Dialogue

pdf bib
CHAT to Your Destination
Fuliang Weng | Baoshi Yan | Zhe Feng | Florin Ratiu | Madhuri Raya | Brian Lathrop | Annie Lien | Sebastian Varges | Rohit Mishra | Feng Lin | Matthew Purver | Harry Bratt | Yao Meng | Stanley Peters | Tobias Scheideck | Badri Raghunathan | Zhaoxia Zhang
Proceedings of the 8th SIGdial Workshop on Discourse and Dialogue

pdf bib
Resolving “You” in Multi-Party Dialog
Surabhi Gupta | John Niekrasz | Matthew Purver | Dan Jurafsky
Proceedings of the 8th SIGdial Workshop on Discourse and Dialogue

pdf bib
Disambiguating Between Generic and Referential “You” in Dialog
Surabhi Gupta | Matthew Purver | Dan Jurafsky
Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions

2006

pdf bib
Unsupervised Topic Modelling for Multi-Party Spoken Discourse
Matthew Purver | Konrad P. Körding | Thomas L. Griffiths | Joshua B. Tenenbaum
Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics

pdf bib
Shallow Discourse Structure for Action Item Detection
Matthew Purver | Patrick Ehlen | John Niekrasz
Proceedings of the Analyzing Conversations in Text and Speech

2005

pdf bib
Meeting Structure Annotation: Data and Tools
Alexander Gruenstein | John Niekrasz | Matthew Purver
Proceedings of the 6th SIGdial Workshop on Discourse and Dialogue

pdf bib
Combining Confidence Scores with Contextual Features for Robust Multi-Device Dialogue
Lawrence Cavedon | Matthew Purver | Florin Ratiu
Proceedings of the Australasian Language Technology Workshop 2005

2004

pdf bib
Incremental Parsing, or Incremental Grammar?
Matthew Purver | Ruth Kempson
Proceedings of the Workshop on Incremental Parsing: Bringing Engineering and Cognition Together

2003

pdf bib
Answering Clarification Questions
Matthew Purver | Patrick G.T. Healey | James King | Jonathan Ginzburg | Greg J. Mills
Proceedings of the Fourth SIGdial Workshop of Discourse and Dialogue

pdf bib
Incremental Generation by Incremental Parsing: Tactical Generation in Dynamic Syntax
Matthew Purver | Masayuki Otsuka
Proceedings of the 9th European Workshop on Natural Language Generation (ENLG-2003) at EACL 2003

2002

pdf bib
Processing Unknown Words in a Dialogue System
Matthew Purver
Proceedings of the Third SIGdial Workshop on Discourse and Dialogue

2001

pdf bib
On the Means for Clarification in Dialogue
Matthew Purver | Jonathan Ginzburg | Patrick Healey
Proceedings of the Second SIGdial Workshop on Discourse and Dialogue

Search
Co-authors