Svetlana Stoyanchev


2023

pdf bib
Evaluating Large Language Models for Document-grounded Response Generation in Information-Seeking Dialogues
Norbert Braunschweiler | Rama Doddipatla | Simon Keizer | Svetlana Stoyanchev
Proceedings of the 1st Workshop on Taming Large Language Models: Controllability in the era of Interactive Assistants!

In this paper, we investigate the use of large language models (LLMs) like ChatGPT for document-grounded response generation in the context of information-seeking dialogues. For evaluation, we use the MultiDoc2Dial corpus of task-oriented dialogues in four social service domains previously used in the DialDoc 2022 Shared Task. Information-seeking dialogue turns are grounded in multiple documents providing relevant information. We generate dialogue completion responses by prompting a ChatGPT model, using two methods: Chat-Completion and LlamaIndex. ChatCompletion uses knowledge from ChatGPT model pre-training while LlamaIndex also extracts relevant information from documents. Observing that document-grounded response generation via LLMs cannot be adequately assessed by automatic evaluation metrics as they are significantly more verbose, we perform a human evaluation where annotators rate the output of the shared task winning system, the two ChatGPT variants outputs, and human responses. While both ChatGPT variants are more likely to include information not present in the relevant segments, possibly including a presence of hallucinations, they are rated higher than both the shared task winning system and human responses.

pdf bib
Proceedings of the 24th Annual Meeting of the Special Interest Group on Discourse and Dialogue
Svetlana Stoyanchev | Shafiq Joty | David Schlangen | Ondrej Dusek | Casey Kennington | Malihe Alikhani
Proceedings of the 24th Annual Meeting of the Special Interest Group on Discourse and Dialogue

2022

pdf bib
Opening up Minds with Argumentative Dialogues
Youmna Farag | Charlotte Brand | Jacopo Amidei | Paul Piwek | Tom Stafford | Svetlana Stoyanchev | Andreas Vlachos
Findings of the Association for Computational Linguistics: EMNLP 2022

Recent research on argumentative dialogues has focused on persuading people to take some action, changing their stance on the topic of discussion, or winning debates. In this work, we focus on argumentative dialogues that aim to open up (rather than change) people’s minds to help them become more understanding to views that are unfamiliar or in opposition to their own convictions. To this end, we present a dataset of 183 argumentative dialogues about 3 controversial topics: veganism, Brexit and COVID-19 vaccination. The dialogues were collected using the Wizard of Oz approach, where wizards leverage a knowledge-base of arguments to converse with participants. Open-mindedness is measured before and after engaging in the dialogue using a questionnaire from the psychology literature, and success of the dialogue is measured as the change in the participant’s stance towards those who hold opinions different to theirs. We evaluate two dialogue models: a Wikipedia-based and an argument-based model. We show that while both models perform closely in terms of opening up minds, the argument-based model is significantly better on other dialogue properties such as engagement and clarity.

pdf bib
Discourse annotation — Towards a dialogue system for pair programming
Cecilia Domingo | Paul Piwek | Svetlana Stoyanchev | Michel Wermelinger
Traitement Automatique des Langues, Volume 63, Numéro 3 : Etats de l'art en TAL [Review articles in NLP]

pdf bib
Combining Structured and Unstructured Knowledge in an Interactive Search Dialogue System
Svetlana Stoyanchev | Suraj Pandey | Simon Keizer | Norbert Braunschweiler | Rama Sanand Doddipatla
Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue

Users of interactive search dialogue systems specify their preferences with natural language utterances. However, a schema-driven system is limited to handling the preferences that correspond to the predefined database content. In this work, we present a methodology for extending a schema-driven interactive search dialogue system with the ability to handle unconstrained user preferences. Using unsupervised semantic similarity metrics and the text snippets associated with the search items, the system identifies suitable items for the user’s unconstrained natural language query. In crowd-sourced evaluation, the users chat with our extended restaurant search system. Based on objective metrics and subjective user ratings, we demonstrate the feasibility of using an unsupervised low latency approach to extend a schema-driven search dialogue system to handle unconstrained user preferences.

2016

pdf bib
Rapid Prototyping of Form-driven Dialogue Systems Using an Open-source Framework
Svetlana Stoyanchev | Pierre Lison | Srinivas Bangalore
Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue

2014

pdf bib
AT&T: The Tag&Parse Approach to Semantic Parsing of Robot Spatial Commands
Svetlana Stoyanchev | Hyuckchul Jung | John Chen | Srinivas Bangalore
Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014)

pdf bib
Dialogue Act Modeling for Non-Visual Web Access
Vikas Ashok | Yevgen Borodin | Svetlana Stoyanchev | IV Ramakrishnan
Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL)

pdf bib
Detecting Inappropriate Clarification Requests in Spoken Dialogue Systems
Alex Liu | Rose Sloan | Mei-Vern Then | Svetlana Stoyanchev | Julia Hirschberg | Elizabeth Shriberg
Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL)

pdf bib
MVA: The Multimodal Virtual Assistant
Michael Johnston | John Chen | Patrick Ehlen | Hyuckchul Jung | Jay Lieske | Aarthi Reddy | Ethan Selfridge | Svetlana Stoyanchev | Brant Vasilieff | Jay Wilpon
Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL)

2013

pdf bib
Exploring Features For Localized Detection of Speech Recognition Errors
Eli Pincus | Svetlana Stoyanchev | Julia Hirschberg
Proceedings of the SIGDIAL 2013 Conference

pdf bib
Modelling Human Clarification Strategies
Svetlana Stoyanchev | Alex Liu | Julia Hirschberg
Proceedings of the SIGDIAL 2013 Conference

2011

pdf bib
Data-oriented Monologue-to-Dialogue Generation
Paul Piwek | Svetlana Stoyanchev
Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies

pdf bib
The CODA System for Monologue-to-Dialogue Generation
Svetlana Stoyanchev | Paul Piwek
Proceedings of the SIGDIAL 2011 Conference

pdf bib
Question Generation Shared Task and Evaluation Challenge – Status Report
Vasile Rus | Brendan Wyse | Paul Piwek | Mihai Lintean | Svetlana Stoyanchev | Cristian Moldovan
Proceedings of the 13th European Workshop on Natural Language Generation

2010

pdf bib
Constructing the CODA Corpus: A Parallel Corpus of Monologues and Expository Dialogues
Svetlana Stoyanchev | Paul Piwek
Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10)

We describe the construction of the CODA corpus, a parallel corpus of monologues and expository dialogues. The dialogue part of the corpus consists of expository, i.e., information-delivering rather than dramatic, dialogues written by several acclaimed authors. The monologue part of the corpus is a paraphrase in monologue form of these dialogues by a human annotator. The annotator-written monologue preserves all information present in the original dialogue and does not introduce any new information that is not present in the original dialogue. The corpus was constructed as a resource for extracting rules for automated generation of dialogue from monologue. Using authored dialogues allows us to analyse the techniques used by accomplished writers for presenting information in the form of dialogue. The dialogues are annotated with dialogue acts and the monologues with rhetorical structure. We developed annotation and translation guidelines together with a custom-developed tool for carrying out translation, alignment and annotation of the dialogues. The final parallel CODA corpus consists of 1000 dialogue turns that are tagged with dialogue acts and aligned with monologue that expresses the same information and has been annotated with rhetorical structure relations.

pdf bib
Harvesting Re-usable High-level Rules for Expository Dialogue Generation
Svetlana Stoyanchev | Paul Piwek
Proceedings of the 6th International Natural Language Generation Conference

pdf bib
The First Question Generation Shared Task Evaluation Challenge
Vasile Rus | Brendan Wyse | Paul Piwek | Mihai Lintean | Svetlana Stoyanchev | Christian Moldovan
Proceedings of the 6th International Natural Language Generation Conference

pdf bib
Generating Expository Dialogue from Monologue: Motivation, Corpus and Preliminary Rules
Paul Piwek | Svetlana Stoyanchev
Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics

2009

pdf bib
Predicting Concept Types in User Corrections in Dialog
Svetlana Stoyanchev | Amanda Stent
Proceedings of SRSL 2009, the 2nd Workshop on Semantic Representation of Spoken Language

pdf bib
Automating Model Building in c-rater
Jana Sukkarieh | Svetlana Stoyanchev
Proceedings of the 2009 Workshop on Applied Textual Inference (TextInfer)

pdf bib
Concept Form Adaptation in Human-Computer Dialog
Svetlana Stoyanchev | Amanda Stent
Proceedings of the SIGDIAL 2009 Conference

pdf bib
Lexical and Syntactic Adaptation and Their Impact in Deployed Spoken Dialog Systems
Svetlana Stoyanchev | Amanda Stent
Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, Companion Volume: Short Papers

pdf bib
Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, Companion Volume: Student Research Workshop and Doctoral Consortium
Ulrich Germann | Chirag Shah | Svetlana Stoyanchev | Carolyn Penstein Rosé | Anoop Sarkar
Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, Companion Volume: Student Research Workshop and Doctoral Consortium

2008

pdf bib
Exact Phrases in Information Retrieval for Question Answering
Svetlana Stoyanchev | Young Chol Song | William Lahti
Coling 2008: Proceedings of the 2nd workshop on Information Retrieval for Question Answering