% This must be in the first 5 lines to tell arXiv to use pdfLaTeX, which is strongly recommended.
\pdfoutput=1
% In particular, the hyperref package requires pdfLaTeX in order to break URLs across lines.

\documentclass[11pt]{article}

% Remove the "review" option to generate the final version.
\usepackage{acl}
%\usepackage[review]{acl}

% Standard package includes
\usepackage{times}
\usepackage{latexsym}
\usepackage{multirow}
\usepackage{url}
\usepackage{hyperref}
\usepackage{graphicx}
%\DeclareEmphSequence{\gtfamily\sffamily\bfseries}
% For proper rendering and hyphenation of words containing Latin characters (including in bib files)
\usepackage[T1]{fontenc}
% For Vietnamese characters
% \usepackage[T5]{fontenc}
% See https://www.latex-project.org/help/documentation/encguide.pdf for other character sets

% This assumes your files are encoded as UTF8
\usepackage[utf8]{inputenc}

% This is not strictly necessary, and may be commented out,
% but it will improve the layout of the manuscript,
% and will typically save some space.
\usepackage{microtype}

\newcommand{\yamada}[1]{\textcolor{magenta}{[Yamada: #1]}}
\newcommand{\bhline}{\noalign{\hrule height 1.2pt}}

% If the title and author information does not fit in the area allocated, uncomment the following
%
%\setlength\titlebox{<dim>}
%
% and set <dim> to something 5cm or larger.

\title{Building a Buzzer-Quiz Answering System}

% Author information can be set in various styles:
% For several authors from the same institution:
% \author{Author 1 \and ... \and Author n \\
%         Address line \\ ... \\ Address line}
% if the names do not fit well on one line use
%         Author 1 \\ {\bf Author 2} \\ ... \\ {\bf Author n} \\
% For authors from different institutions:
% \author{Author 1 \\ Address line \\  ... \\ Address line
%         \And  ... \And
%         Author n \\ Address line \\ ... \\ Address line}
% To start a seperate ``row'' of authors use \AND, as in
% \author{Author 1 \\ Address line \\  ... \\ Address line
%         \AND
%         Author 2 \\ Address line \\ ... \\ Address line \And
%         Author 3 \\ Address line \\ ... \\ Address line}

\author{
  Naoya Sugiura\hspace{0.5em}
  Kosuke Yamada\hspace{0.5em}
  Ryohei Sasano\hspace{0.5em}
  Koichi Takeda\hspace{0.5em} 
  Katsuhiko Toyama\hspace{0.5em} \\
  Graduate School of Informatics, Nagoya University, Japan\\
  \texttt{\{sugiura.naoya.e7,yamada.kosuke.v1\}@s.mail.nagoya-u.ac.jp} \\
  \texttt{\{sasano,takedasu,toyama\}@i.nagoya-u.ac.jp} \\
}
  
\begin{document}
\maketitle
\begin{abstract}
A buzzer quiz is a genre of quiz in which multiple players simultaneously listen to a quiz being read aloud and respond it by buzzing in as soon as they can predict the answer.
Because incorrect answers often result in penalties, a buzzer-quiz answering system must not only predict the answer from only a part of a question but also estimate the accuracy of the answer.
In this paper, we introduce two types of buzzer-quiz answering systems: (1) a system that directly generates an answer from part of a question by using an autoregressive language model; and (2) a system that first reconstructs the entire question by using an autoregressive language model and then determines the answer according to the reconstructed question.
We then propose a method to estimate the accuracy of the answers for each system by using the internal scores of each model. %and we validate the effectiveness of our approach through experiments.

%In this study, we aim to develop a buzzer quiz answering system that can output appropriate answers to incomplete quiz questions.
%Specifically, we will build two systems: one that directly generates answers after the question using GPT, and the other uses GPT to complete the question and then generates answers by the retriever-reader approach with DPR.
%In these buzzer quiz answering systems, it is crucial to assess the plausibility of the answer at each point in time. We also propose a method for calculating confidence score, which serve as an indicator for this purpose. As a result, by answering only questions with high confidence score, we achieved a high accuracy rate even when the overall accuracy was low.


\end{abstract}

\section{Introduction}
We use the term ``buzzer quiz'' to refer to a genre of quiz which questioner reads quiz questions aloud and players answer by buzzing in as soon as they can predict the answer.
A well-known example of what we call a buzzer quiz here is the U.S. TV program \emph{Jeopady!}, in which contestants must buzz in with a lock-out device before trying to answer a question.
%A buzzer quiz is a genre of quiz in which players answer by buzzing as soon as they can predict an answer, while the questioner reading aloud the quiz text. 
Because of the importance of buzzing in quickly, players normally answer incomplete questions.
Buzzer quizzes have been widely studied as open-domain question answering (QA) tasks because they do not limit the scope of knowledge.
However, the major datasets for open-domain QA tasks, like Natural Questions \cite{kwiatkowski-etal-2019-natural} and TriviaQA \cite{joshi-etal-2017-triviaqa} cointain complete questions. 
Consequently, systems built using those datasets \cite{karpukhin-etal-2020-dense,yamada-etal-2021-efficient,izacard-grave-2021-leveraging} are not designed to answer incomplete questions.

Furthermore, it is certainly crucial in buzzer quizzes to give correct answers, but it is also essential to consider the plausibility of a predicted answer based on the given question at that moment and to decide whether to actually respond.
For example, consider the question listed in Table \ref{tab:example} if it has not been read past the phrase ``200-hit.''
At that point, because other baseball players also hold records comparable to that of Pete Rose, it is difficult to narrow the answer down to a single candidate.
This makes the predicted answer at that moment more likely to be incorrect, so it would be better not to answer at that point.
On the other hand, once the question has been read further, the predicted answer converges to the correct answer, ``Ichiro Suzuki.''
Hence, to construct a more effective buzzer-quiz answering system, we need an indicator of a predicted answer's liklihood of being correct, which call a ``confidence score.''

%Therefore, it is conceivable that a more effective buzzer quiz answering system can be constructed by calculating the confidence score for the candidate answers at each point in the question.
In addition, the capability to respond to buzzer quizzes by answering incomplete questions could help replicate the human capacity to smoothly generate responses in a conversation by sequentially predicting the content of the dialogue.

\begin{table}[t!]
\centering
\small
\begin{tabular}{@{\ }l@{\ }} \bhline
\rule{0pt}{2.5ex}
\textbf{Q (75\% completeness):} Pete Rose and this player are \\
tied with ten 200-hit seasons each. 
This Japanese \\
outfielder played most of his career with the Mar\textcolor[gray]{0.7}{iners,} \\
\textcolor[gray]{0.7}{and currently plays for the Marlins.} \\
\rule{0pt}{2.5ex}
\textbf{Confidence score:} 0.991 \textbf{A:} Ichiro Suzuki \ \ \emph{\color{red} correct} \\ \bhline

\rule{0pt}{2.5ex}
\textbf{Q (25\% completeness)}: Pete Rose and this player are \\
tied with ten 2\textcolor[gray]{0.7}{00-hit seasons each. This Japanese} \\
\textcolor[gray]{0.7}{outfielder played most
of his career with the Mariners,} \\
\textcolor[gray]{0.7}{and currently plays for the Marlins.} \\
\rule{0pt}{2.5ex}
\textbf{Confidence score:} 0.125 \textbf{A:} Ty Cobb \ \ \emph{\color{blue} incorrect} \\ \bhline
\rule{0pt}{2.5ex}
\end{tabular}
\caption{Output example. Black text indicates the portion of the question text that has been read, while gray text indicates the unread portion. Furthermore, ``completeness'' denotes the percentage of the question text that has been read, and the ``confidence score'' refers to a value indicating the likelihood of the predicted answer being correct.}
\label{tab:example}
\end{table}


In this study, we first constructed a buzzer-quiz answering system that produces appropriate answers for incomplete questions, and we propose the methods of calculating the confidence scores for two different models.
Specifically, we constructed two systems: the GPT (Inference) system, which directly generates answers in response to a question by using GPT \cite{radford-2018-improving}; and the GPT + DPR system, which generates answers through a retriever-reader approach using Dense Passage Retrieval (DPR) \cite{karpukhin-etal-2020-dense}, after completing the question via GPT. 
For the former system, we calculate a confidence score by using token output probabilities during answer generation, while for the latter system, we use scores that are used in the output of the model. 
The confidence scores given in Table \ref{tab:example} are scores from the GPT (Inference) model. 
When the confidence score is low, there are various possible answers because the question's completeness is low. 
In contrast, when more information about the correct answer is provided, the confidence score becomes higher. 
This shows that the confidence scores works well in s buzzer-quiz answering system.




\section{Proposed Method}
\label{sec:DPRandGPT}
% We first introduce related research on open-domain question answering task, which is basis of the proposed models. 
% Then, we propose a buzzer quiz answering system that answers incomplete question sentences and a method for calculating confidence score based on the output of the model.
We propose two types of buzzer-quiz answering systems based on open-domain question answering systems. 
We also propose methods to estimate the accuracy of the answers in each system by using the internal scores in each model.

\subsection{Open-Domain Question Answering System}
\label{sec:OpenDomain}
% A quiz consists of fasctoid questions without domain restrictions, and answering these questions is a kind of open-domain QA tasks.
% In open-domain QA, there are two mainstream approaches to answering models: a single model that directly generates answers and a retriever-reader approach that answers by combining two tasks, searching for relevant documents and extracting answer.
% GPT \cite{radford-2018-improving} is a representative model of a single model for generating answers.
% GPT is a pre-trained language model based on the Transformer decoder \cite{vaswani-2017-attention} that learned to predict subsequent words in a given sentence by a large corpus of text.
Because a quiz comprises factoid questions without domain restrictions, the buzzer-quiz answering task is a type of open-domain QA task.
In open-domain QA, there are two mainstream approaches.

The first is a generation-based approach that generates answers directly in response to input questions.
A representative models is GPT \cite{radford-2018-improving}, which is a pretrained language model that is based on the Transformer decoder \cite{vaswani-2017-attention} and is trained to predict word sequences from a context by using a large text corpus.
Because of this property, GPT can be used in language generation tasks that involve generating text in response to input text.
In the case of QA, GPT can generate answers by formatting the input in such a way as to infer only the answer to a question.
Furthermore, because GPT often achieves higher performance through finetuning with datasets from downstream tasks, such finetuning can be applied to build QA models.

% A representative example of an answering model for compound tasks is the retriever-reader model based on DPR. 
% The retriever employs a dual-encoder structure using different BERT \cite{devlin-2019-bert} models for the question text and each document. 
% When inputting sentences into BERT, a special token \verb|[CLS]| is inserted at the beginning of the document, and the embedding representations for the question text and each document are obtained.
% And then this method selects documents based on semantic similarity calculated by the inner product of the obtained representations \cite{karpukhin-etal-2020-dense}.
% In the reader, BERT predicts the relevant documents containing the correct answer and extracts the answer portion within the document. 
% For the related document prediction, it predicts the document likely to contain the answer based on the \verb|[CLS]| token position. 
% Then, for the chosen document, it performs the answer portion extraction prediction, determining the starting and ending points of the token sequence that forms the answer and outputs it.
The second major open-domain QA approach is a retriever-reader approach that searches for documents related to a question and extracts the answer from the documents.
A representative model is the retriever-reader model, which uses DPR as the retriever.
DPR uses a dual encoder network with different BERT models \cite{devlin-2019-bert} for questions and documents.
When sentences are input to BERT, a special token \verb|[CLS]| is inserted at the beginning of a document, and the embedding representations for the question text and each document are obtained.
Then, documents are selected according to the semantic similarity calculated as the inner product of the obtained representations \cite{karpukhin-etal-2020-dense}.
In the reader, BERT predicts the relevant documents containing the correct answer and extracts the answer portion within a document. 
Specifically, it predicts the document that is most likely to contain the answer at the position of the token \verb|[CLS]|. 
Then, it performs the answer-portion extraction from for the predicted document and determines the start and end points of the token sequence that forms the answer.

\subsection{Buzzer-Quiz Answering Systems}
The effectiveness of the open-domain QA systems that answer complete questions has been confirmed, but their effectiveness for a buzzer-quiz answering system remains unclear, because such a system requires to answer incomplete questions.
Generally, when only a part of a question is given, the nature of the problem differs significantly from the case of a complete question, because there may be multiple possible answers, or the necessary information to determine the answer might not be available yet.
% Therefore, it is needed that a specific method for buzzer quiz.

In this study, we constructed two buzzer-quiz answering systems: one that relies solely on inference via GPT, called the \textbf{GPT (Inference)} system, and another that uses GPT for question completion and applies the retriever-reader approach with DPR, called the \textbf{GPT + DPR} system.
For the GPT (Inference) system, the designed input format is ``[question text] + `/the answer is','' which prompts the model to generate the answer within the single quotation marks, which is then used as the predicted answer.
The propose of inserting a slash `/' between the question text and ``the answer is'' is to make the model to recognize the boundary of the question text, which prevents  completion of incomplete questions.
For the GPT + DPR system, an incomplete question is input to the GPT to complete the question text, and the resulting complete question is then used as input for the DPR-based retriever-reader model to generate the answer. 
This approach combines the strengths of GPT for text completion and the effectiveness of DPR for retrieving documents and extracting answers.

\subsection{Confidence Scores}
\label{sec:ConfidenceScore}
Next, we propose to calculate the confidence scores for predicted answers by using the internal scores that each model uses when it generates their outputs for the buzzer-quiz answering system.
Here, the confidence score means an indicator for judging whether a predicted answer is correct.
For higher values of our proposed confidence scores, we expect a higher percentage of correct answers.

For the GPT (Inference) model, we use the generation probability of the first token in the predicted answer (referred to as the \textbf{generation score}) as the confidence score.
When given a sentence's first $n$ tokens during sentence completion, GPT outputs the $(n+1)$-th token from the vocabulary with the highest generation score.
The first token largely determines a direction of the answer in buzzer quiz, because the answer is often comprises a small number of tokens. 
Hence, we adopt only first token's generation score as the confidence score.

In the GPT+DPR model, we consider three internal scores that can be used as confidence scores: the \textbf{document score} and the \textbf{extraction score} calculated by the reader, as well as their arithmetic mean, the \textbf{average score}.
In the reader, each \verb|[CLS]| token int the document is scored through a learned linear layer, and the document with the highest score is selected. 
This score is the document score.
Then, the model extracts the span that contains the answer from the selected document by calculating the span score, which is composed of a start score and an end score.
The extraction score is the sum of these start and end scores.

\section{Experiments}
We conducted two experiments: an accuracy verification that aims to evaluate of the accuracy of the proposed buzzer quiz answering system and an investigation of confidence score effectiveness that aims to examine the effectiveness of the confidence scores for each model.
We define the question completeness as $x$\% when the question is truncated at the first $x$\% of the text based on the character count.
In the accuracy verification, we applied the GPT (inference) model and the GPT+DPR model to problems with question completeness of 25\%, 50\%, 75\%, and 100\%, and verified their accuracy.
In the investigation of confidence score effectiveness, we evaluated the confidence scores of each model explained in Section \ref{sec:ConfidenceScore} by examining the relationship between the confidence scores and the accuracy rates at each question completeness.


\subsection{Settings}
\begin{table}[t!]
\centering
\small
\begin{tabular}{cc|rr} \bhline
Subset & Source & \multicolumn{1}{|c}{Size} & \multicolumn{1}{c}{Length} \\ \bhline
\multirow{2}{*}{Train} & AIO & 17,735 &  48.2\\
& Minhaya & 35,149 & 64.8 \\ \hline
Dev & AIO & 1,000 & 46.9 \\ \hline
Test & AIO & 2,000 & 51.6 \\ \bhline
\end{tabular}
\caption{Overview of the dataset. ``Length'' means the average number of characters of questions in the dataset.}
\label{tab:data}
\end{table}

\paragraph{Dataset}
In our experiments, we used the 2nd AIO Official Dataset (AIO)\footnote{\url{https://sites.google.com/view/project-aio/dataset}}, which contains past questions of the Japanese quiz competitions, and we crawled past questions of the Japanese quiz application ``Minna de Hayaoshi Quiz'' (Minhaya)\footnote{\url{https://livequiz.work/minhaya1/}}.
Table \ref{tab:data} shows the number of quiz-answer pairs and the average number of character of the questions for each dataset.
Note that for training DPR, in addition to the question and answer, positive and negative documents are required. 
Therefore, DPR was trained using only the AIO dataset, while the additional ``Minhaya'' dataset was used only for training GPT.

\paragraph{Comparison Models}
We compared the two models, GPT (Inference) and GPT+DPR in accuracy verification.
In the investigation of confidence effectiveness, for GPT (inference), we used the generation score, while for GPT+DPR, we conducted experiments using three different scores: related document score, extraction score, and average score.

We used the Japanese GPT model on Hugging Face \cite{wolf-2020-transformers}\footnote{\url{https://huggingface.co/rinna/japanese-gpt-1b}} and the DPR model based on Japanese BERT-large\footnote{\url{https://huggingface.co/cl-tohoku/bert-large-japanese}} learned by Japanese Wikipedia corpus\footnote{\url{https://github.com/cl-tohoku/AIO2_DPR_baseline}}.
For GPT (inference), we fine-tuned the model using the training set formatted as ``[question text] + '/ the answer is' [answer]''. 
Then GPT+DPR, GPT was fine-tuned using only the question from the training set. 
In both cases, the training was conducted for 5 epochs.
DPR is based on Japanese BERT\footnote{\url{https://huggingface.co/cl-tohoku/bert-large-japanese}} for both retriever and reader components. 
The retriever was trained with a batch size of 128, a learning rate of 1e-5, and for 5 epochs, while the reader was trained with a batch size of 8, a learning rate of 2e-5, and for 3 epochs.

\paragraph{Metrics}
In the accuracy verification, the correctness of the predicted answer was assessed based on exact matching.
%However, to prevent incorrect answers due to variations in notation, we removed six types of symbols - "(", ")", "[", "]", "・", and "=" - from the predicted answer before comparing.
In the investigation of confidence score effectiveness, we created a correct answer rate - answer generating rate curve and investigated the effectiveness of the confidence by the area under the curve (AUC). 
Here, the answer generating rate refers to the proportion of times the system actually provides an answer.
If the models only answer the question whose confidence score exceeds the threshold $\alpha$, we can control the answer rate by changing $\alpha$. 
On the other hand, the correct answer rate is the proportion of correct answers among those output by the models.
When $\alpha$ is set to a value less than 0, it coincides with the overall correct answer rate of the system. 
As $\alpha$ increases, only questions with high confidence scores will be answered, so the correct answer rate is expected to increase.

\subsection{Accuracy Verification}
\label{sec:experimentA}

\begin{table}[t!]
\centering
\small
\begin{tabular}{llcccc} \bhline
Model & 25\% & 50\% & 75\% & 100\% \\ \bhline
GPT (inference) & 11.9 & 27.9 & 45.6 & 56.2 \\
GPT+DPR & 11.9 & 28.8 & 45.9 & 62.0 \\ \bhline
\end{tabular}
\caption{Results of accuracy verification. The x\% represents the question completeness.}
\label{tab:EM}
\end{table}

Table \ref{tab:EM} shows the accuracy rates for GPT (inference) and GPT+DPR based on each question completeness.
As the question completeness decreased, the correct answer rate also decreased. 
However, the rate of decrease was not proportional to the question completeness. 
From 100\% to 75\%, the decline was relatively gentle.
This is likely because many important words that determine the answer appear in the first half to the middle of the quiz question text, whereas cases where information-rich words appear in the latter half are relatively rare.
Comparing the scores of the two models, GPT+DPR performed better when the question completeness was 100\%. 
However, when the question was incomplete, no significant difference in performance between the two models was observed.


\subsection{Investigation of Confidence Score Effectiveness}
\label{sec:experimentB}

\begin{table}[t!]
\centering
\small
\vspace{0.5ex}
\begin{tabular}{@{\ }l@{\ \ }c@{\ \ }c@{\ \ }c@{\ \ }c@{\ \ }c@{\ }} \bhline
Model & Scoring & 25\% & 50\% & 75\% & 100\% \\ \bhline
GPT (inference) & generation score & \textbf{41.4} & \textbf{63.6} & \textbf{81.2} & \textbf{85.9} \\ \hline
& document score & 31.8 & 58.1 & 77.3 & 84.8 \\
GPT+DPR & extraction score & 25.0 & 51.3 & 70.0 & 84.1 \\
& average score & 29.1 & 56.1 & 75.8 & 84.0 \\
\bhline
\end{tabular}
\caption{AUC for each question completeness. ``Scoring'' means the internal scores we used.}
\label{tab:AUC}
\end{table}

\begin{figure}[t!]
\centering
\includegraphics[width=\linewidth]{PRC_Eng.pdf}
\vspace{-4ex}
\caption{Correct answer rate - answer generating rate curve}
\label{fig:AUC}
\end{figure}

Table \ref{tab:AUC} shows the AUC for each question completeness.
Among the three confidence measures for GPT+DPR, using the related document score resulted in the highest AUC.
Furthermore, in all the results, the generation score of GPT (inference) achieved the highest AUC.
Next, using the related document score, which had the highest AUC in GPT+DPR, we compared the correct answer rate - answer rate curves of the GPT (inference) model and the GPT+DPR model.
Figure \ref{fig:AUC} shows the results.
In all settings, it can be recognized that the accuracy increases by narrowing down the questions to be answered only by those with high confidence scores, confirming the effectiveness of the confidence scores.
When comparing GPT (inference) and GPT+DPR, as shown in Table \ref{tab:EM}, the accuracy at an answer rate of 1.0 is higher for GPT+DPR in the case where a question completeness is 100\%, and equivalent in other cases. 
However, when an answer rate less than 0.8, GPT (inference) had a higher accuracy in all cases.
This difference is more obvious when the question completeness and the answer rate are low. 
For example, in the setting with a question completeness of 25\% and an answer rate of 0.1, the accuracy of GPT+DPR is around 0.5, while the accuracy of GPT (inference) is around 0.8, showing a significant difference.
Based on this, it can be said that GPT (inference) is more suitable for quick-answer quizzes.




\section{Conclusion}
In this study, we constructed two systems for answering buzzer quiz questions, which were not considered in previous research: GPT (inference) and GPT+DPR. 
Then we evaluated the accuracy for each question completeness. 
Furthermore, we investigated the relationship between confidence score and accuracy, and verified the validity of using the internal scores of the models as confidence scores.



% Entries for the entire Anthology, followed by custom entries
\bibliography{reference}
\bibliographystyle{acl_natbib}



\end{document}
