
\pdfoutput=1


\documentclass[11pt]{article}


\usepackage{acl}



\usepackage{times}
\usepackage{latexsym}
\usepackage{multirow}
\usepackage{url}
\usepackage{hyperref}
\usepackage{graphicx}

\usepackage[T1]{fontenc}

\usepackage[utf8]{inputenc}


\usepackage{microtype}

\newcommand{\yamada}[1]{\textcolor{magenta}{[Yamada: #1]}}
\newcommand{\bhline}{\noalign{\hrule height 1.2pt}}

\title{Building a Buzzer Quiz Answering System}


\author{
  Naoya Sugiura\hspace{0.5em}
  Kosuke Yamada\hspace{0.5em}
  Ryohei Sasano\hspace{0.5em}
  Koichi Takeda\hspace{0.5em} 
  Masahiko Toyama\hspace{0.5em} \\
  \texttt{\{sugiura.naoya.e7,yamada.kosuke.v1\}@s.mail.nagoya-u.ac.jp} \\
  \texttt{\{sasano,takedasu,toyama\}@i.nagoya-u.ac.jp}
}
  
\begin{document}
\maketitle
\begin{abstract}
Buzzer quiz is a genre of quiz in which multiple players simultaneously listen to a quiz being read aloud and answer it by pressing the buzzer as soon as they can predict the answer.
Since incorrect answers often result in penalties, a buzzer quiz answering system must be able to not only predict the answer using only a part of the question text but also estimate the accuracy of the answer.
In this paper, we introduce two types of buzzer quiz answering systems: 1) a system that directly generates an answer using an autoregressive language model from a part of the question, and 2) a system that first reconstructs the entire question using an autoregressive language model and then determines the answer based on the reconstructed question text. 
We then propose a method to estimate the accuracy of the answer for each system by utilizing the internal score of each model. 


\end{abstract}

\section{Introduction}
A buzzer quiz is a genre of quiz in which players answer by buzzing as soon as they can predict an answer, while the questioner reading aloud the quiz text. 
The importance of buzzer quizzes is rapidness of buzzing, so players normally answer to incomplete questions.
Buzzer quizzes have been widely studied as open-domain question answering (QA) tasks because they do not limit the scope of knowledge.
However, major datasets of open-domain QA tasks like Natural Questions \cite{kwiatkowski-etal-2019-natural} and TriviaQA \cite{joshi-etal-2017-triviaqa} consist of complete question sentences. 
Consequently, systems built using these datasets \cite{karpukhin-etal-2020-dense,yamada-etal-2021-efficient,izacard-grave-2021-leveraging} are not designed to answer incomplete questions.

Furthermore, in buzzer quizzes, it is certainly crucial to providing an correct answer, but it is also essential to consider the plausibility of the predicted answer based on the given question at that moment and decide whether to actually respond or not.
For example, consider a question as shown in Table \ref{tab:example} when it has not been read up to ``200-hit.''
At this point, as there are other baseball players who hold records comparable to Pete Rose, it is difficult to narrow down the answer to a single candidate.
This makes the predicted answer at that moment more likely to be incorrect, so we shouldn't answer at this time.
On the other hand, if the question is read further, the predicted answer converges to the correct answer, ``Ichiro Suzuki.''
Therefore, to construct more effective buzzer quiz answering system, An indicator of the likelihood of the predicted answer being correct is needed.
we named it confidence score.


Finally, the ability to respond to buzzer quizzes, answering to incomplete questions, could contribute to replicating the human capacity to smoothly generate responses in a conversation by sequentially predicting the content of the dialogue.

\begin{table}[t!]
\centering
\small
\begin{tabular}{@{\ }l@{\ }} \bhline
\rule{0pt}{2.5ex}
\textbf{Q (75\% completeness):} Pete Rose and this player are \\
tied with ten 200-hit seasons each. 
This Japanese \\
outfielder played most of his career with the Mar\textcolor[gray]{0.7}{iners,} \\
\textcolor[gray]{0.7}{and currently plays for the Marlins.} \\
\rule{0pt}{2.5ex}
\textbf{Confidence score:} 0.991 \textbf{A:} Ichiro Suzuki \ \ \emph{\color{red} correct} \\ \bhline

\rule{0pt}{2.5ex}
\textbf{Q (25\% completeness)}: Pete Rose and this player are \\
tied with ten 2\textcolor[gray]{0.7}{00-hit seasons each. This Japanese} \\
\textcolor[gray]{0.7}{outfielder played most
of his career with the Mariners,} \\
\textcolor[gray]{0.7}{and currently plays for the Marlins.} \\
\rule{0pt}{2.5ex}
\textbf{Confidence score:} 0.125 \textbf{A:} Ty Cobb \ \ \emph{\color{blue} incorrect} \\ \bhline
\rule{0pt}{2.5ex}
\end{tabular}
\caption{An output example. Black text indicates the portion of the question text that has been read, while gray text represents the unread portion. Furthermore, ``completeness'' denotes the percentage of the question text that has been read, and the ``confidence score'' refers to the value output by the model, indicating the likelihood of the predicted answer being correct.}
\label{tab:example}
\end{table}


In this study, we first construct a buzzer quiz answering system that produces appropriate answers for incomplete questions, and we propose the method of calculating the confidence score for each model.
Specifically, we construct two systems: GPT (inference) that generates answers directly following the question text using GPT \cite{radford-2018-improving}, and GPT+DPR that generates answers through a retriever-reader approach using Dense Passage Retrieval (DPR) \cite{karpukhin-etal-2020-dense} after completing the question text with GPT. 
For the former system, we calculate confidence score using token output probabilities during answer generation, while for the latter system, we use scores employed in the output of the model. 
The confidence scores shown in Table \ref{tab:example} represent the scores from GPT (inference). 
When the confidence score is low, there are various possible answers because the completeness of the question is low, whereas more information of correct answer is provided, the confidence score become higher. 
This shows that the confidence score works well in the buzzer quiz answering system.




\section{Proposed Method}
\label{sec:DPRandGPT}

We propose two types of buzzer quiz answering systems based on open-domain question answering systems, and then methods to estimate the accuracy of the answer for each system by utilizing the internal score of each model.

\subsection{Open-domain Question Answering System}
\label{sec:OpenDomain}

Because a quiz consists of factoid questions without domain restrictions, the buzzer quiz answering task is a type of open-domain QA task.
In open-domain QA, there are two mainstream approaches.

One is a generation-based approach that inputs questions and generates answers directly.
One of the representative models is GPT \cite{radford-2018-improving}.
The model is a pre-trained language model based on the Transformer decoder \cite{vaswani-2017-attention} that trained to predict word sequences following a context using a large text corpus.
Because of its property, GPT can be used in language generation tasks that involve generating text following the input text.
In the case of question-answering, GPT can generate answers by formatting the input in such a way that gets it infer only the answer to the question.
Furthermore, because GPT is often able to achieve higher performance by fine-tuning using datasets from downstream tasks, fine-tuning is performed to build question-answering models.


Another is a retriever-reader approach that searches for documents related to the question and extracts the answer from documents.
One of the representative methods is the retriever-reader model, which uses DPR as the retriever.
DPR employs a dual encoder network with different BERTs \cite{devlin-2019-bert} for questions and documents.
When inputting sentences into BERT, a special token \verb|[CLS]| is inserted at the beginning of the document, and the embedding representations for the question text and each document are obtained.
And then this method selects documents based on semantic similarity calculated by the inner product of the obtained representations \cite{karpukhin-etal-2020-dense}.
In the reader, BERT predicts the relevant documents containing the correct answer and extracts the answer portion within the document. 
For the related document prediction, it predicts the document that are most likely to contain the answer at the position of the token \verb|[CLS]|. 
Then, it performs the answer portion extraction prediction for the predicted document and determines the starting and ending points of the token sequence that forms the answer.

\subsection{Buzzer Quiz Answering System}
The effectiveness of the open-domain QA system that answers to the full question text, has been confirmed, but their effectiveness of the buzzer quiz answering system remains unclear because the system requires to answer to a part of a question text.
Generally, when only a part of a question is given, the nature of the problem differs significantly from that of a complete question, because there may be multiple possible answers, or the necessary information to determine the answer might not be given.

In this study, we construct two buzzer quiz systems: one that relies solely on inference using GPT, referred to as the \textbf{GPT (Inference)} system, and another that utilizes GPT for question completion and employs the retriever-reader approach with DPR, called the \textbf{GPT+DPR} system.
In the GPT (Inference) system, the input format is designed as ``[question text] + '/the answer is' '', which prompts the model to generate the answer within the quotation marks, and then the content inside the quotation marks is used as the predicted answer.
Inserting a `/' between the question text and ``the answer is'' aims to make the model to recognize the boundary of the question text, which prevents incomplete question completion.
In the GPT+DPR system, the incomplete sentence is input into GPT to complete the question text, and the resulting complete sentence is then used as input for the DPR-based retriever-reader model to generate the answer. 
This approach combines the strengths of GPT for text completion and the effectiveness of DPR for retrieving documents and extracting answers.

\subsection{Confidence Scores}
\label{sec:ConfidenceScore}
We propose to calculate the confidence score for predicted answers using the internal scores that are utilized by each model when generating their outputs for the buzzer quiz answering system.
The confidence score means an indicator for judging whether a predicted answer is correct or not.
With our proposed confidence score, it is expected that the higher the value, the higher the percentage of correct answers.

In the GPT (inference) model, we use the generation probability of the first token in the predicted answer (referred to as the \textbf{generation score}) as the confidence score.
During sentence completion, GPT outputs the $(n+1)$-th token from the vocabulary with the highest generation score when provided with the first $n$ tokens of a sentence.
The first token largely determines the direction of the answer in buzzer quiz because the answer is often composed of a small number of tokens. 
Therefore, we adopt only the generation score of the first token as the confidence score.

In the GPT+DPR model, we consider three internal scores that can be used as confidence scores: the \textbf{document score} and the \textbf{extraction score} calculated by the reader, as well as their arithmetic mean, the \textbf{average score}.
In the reader, each \verb|[CLS]| token int the document is scored through a learned linear layer, and the document with the highest score is selected. 
This score is the document score.
Then, the model extracts the span that contains the answer from the selected document by calculating the span score, which is composed of a start score and an end score.
The extraction score is the sum of these start and end scores.

\section{Experiments}
We conducted two experiments: an accuracy verification that aims to evaluate of the accuracy of the proposed buzzer quiz answering system and an investigation of confidence score effectiveness that aims to examine the effectiveness of the confidence scores for each model.
We define the question completeness as $x$\% when the question is truncated at the first $x$\% of the text based on the character count.
In the accuracy verification, we applied the GPT (inference) model and the GPT+DPR model to problems with question completeness of 25\%, 50\%, 75\%, and 100\%, and verified their accuracy.
In the investigation of confidence score effectiveness, we evaluated the confidence scores of each model explained in Section \ref{sec:ConfidenceScore} by examining the relationship between the confidence scores and the accuracy rates at each question completeness.


\subsection{Settings}
\begin{table}[t!]
\centering
\small
\begin{tabular}{cc|rr} \bhline
Subset & Source & \multicolumn{1}{|c}{Size} & \multicolumn{1}{c}{Length} \\ \bhline
\multirow{2}{*}{Train} & AIO & 17,735 &  48.2\\
& Minhaya & 35,149 & 64.8 \\ \hline
Dev & AIO & 1,000 & 46.9 \\ \hline
Test & AIO & 2,000 & 51.6 \\ \bhline
\end{tabular}
\caption{Overview of the dataset. ``Length'' means the average number of characters of questions in the dataset.}
\label{tab:data}
\end{table}

\paragraph{Dataset}
In our experiments, we used the 2nd AIO Official Dataset (AIO)\footnote{\url{https://sites.google.com/view/project-aio/dataset}}, which contains past questions of the Japanese quiz competitions, and we crawled past questions of the Japanese quiz application ``Minna de Hayaoshi Quiz'' (Minhaya)\footnote{\url{https://livequiz.work/minhaya1/}}.
Table \ref{tab:data} shows the number of quiz-answer pairs and the average number of character of the questions for each dataset.
Note that for training DPR, in addition to the question and answer, positive and negative documents are required. 
Therefore, DPR was trained using only the AIO dataset, while the additional ``Minhaya'' dataset was used only for training GPT.

\paragraph{Comparison Models}
We compared the two models, GPT (Inference) and GPT+DPR in accuracy verification.
In the investigation of confidence effectiveness, for GPT (inference), we used the generation score, while for GPT+DPR, we conducted experiments using three different scores: related document score, extraction score, and average score.

We used the Japanese GPT model on Hugging Face \cite{wolf-2020-transformers}\footnote{\url{https://huggingface.co/rinna/japanese-gpt-1b}} and the DPR model based on Japanese BERT-large\footnote{\url{https://huggingface.co/cl-tohoku/bert-large-japanese}} learned by Japanese Wikipedia corpus\footnote{\url{https://github.com/cl-tohoku/AIO2_DPR_baseline}}.
For GPT (inference), we fine-tuned the model using the training set formatted as ``[question text] + '/ the answer is' [answer]''. 
Then GPT+DPR, GPT was fine-tuned using only the question from the training set. 
In both cases, the training was conducted for 5 epochs.
DPR is based on Japanese BERT\footnote{\url{https://huggingface.co/cl-tohoku/bert-large-japanese}} for both retriever and reader components. 
The retriever was trained with a batch size of 128, a learning rate of 1e-5, and for 5 epochs, while the reader was trained with a batch size of 8, a learning rate of 2e-5, and for 3 epochs.

\paragraph{Metrics}
In the accuracy verification, the correctness of the predicted answer was assessed based on exact matching.
In the investigation of confidence score effectiveness, we created a correct answer rate - answer generating rate curve and investigated the effectiveness of the confidence by the area under the curve (AUC). 
Here, the answer generating rate refers to the proportion of times the system actually provides an answer.
If the models only answer the question whose confidence score exceeds the threshold $\alpha$, we can control the answer rate by changing $\alpha$. 
On the other hand, the correct answer rate is the proportion of correct answers among those output by the models.
When $\alpha$ is set to a value less than 0, it coincides with the overall correct answer rate of the system. 
As $\alpha$ increases, only questions with high confidence scores will be answered, so the correct answer rate is expected to increase.

\subsection{Accuracy Verification}
\label{sec:experimentA}

\begin{table}[t!]
\centering
\small
\begin{tabular}{llcccc} \bhline
Model & 25\% & 50\% & 75\% & 100\% \\ \bhline
GPT (inference) & 11.9 & 27.9 & 45.6 & 56.2 \\
GPT+DPR & 11.9 & 28.8 & 45.9 & 62.0 \\ \bhline
\end{tabular}
\caption{Results of accuracy verification. The x\% represents the question completeness.}
\label{tab:EM}
\end{table}

Table \ref{tab:EM} shows the accuracy rates for GPT (inference) and GPT+DPR based on each question completeness.
As the question completeness decreased, the correct answer rate also decreased. 
However, the rate of decrease was not proportional to the question completeness. 
From 100\% to 75\%, the decline was relatively gentle.
This is likely because many important words that determine the answer appear in the first half to the middle of the quiz question text, whereas cases where information-rich words appear in the latter half are relatively rare.
Comparing the scores of the two models, GPT+DPR performed better when the question completeness was 100\%. 
However, when the question was incomplete, no significant difference in performance between the two models was observed.


\subsection{Investigation of Confidence Score Effectiveness}
\label{sec:experimentB}

\begin{table}[t!]
\centering
\small
\vspace{0.5ex}
\begin{tabular}{@{\ }l@{\ \ }c@{\ \ }c@{\ \ }c@{\ \ }c@{\ \ }c@{\ }} \bhline
Model & Scoring & 25\% & 50\% & 75\% & 100\% \\ \bhline
GPT (inference) & generation score & \textbf{41.4} & \textbf{63.6} & \textbf{81.2} & \textbf{85.9} \\ \hline
& document score & 31.8 & 58.1 & 77.3 & 84.8 \\
GPT+DPR & extraction score & 25.0 & 51.3 & 70.0 & 84.1 \\
& average score & 29.1 & 56.1 & 75.8 & 84.0 \\
\bhline
\end{tabular}
\caption{AUC for each question completeness. ``Scoring'' means the internal scores we used.}
\label{tab:AUC}
\end{table}

\begin{figure}[t!]
\centering
\includegraphics[width=\linewidth]{PRC_Eng.pdf}
\vspace{-4ex}
\caption{Correct answer rate - answer generating rate curve}
\label{fig:AUC}
\end{figure}

Table \ref{tab:AUC} shows the AUC for each question completeness.
Among the three confidence measures for GPT+DPR, using the related document score resulted in the highest AUC.
Furthermore, in all the results, the generation score of GPT (inference) achieved the highest AUC.
Next, using the related document score, which had the highest AUC in GPT+DPR, we compared the correct answer rate - answer rate curves of the GPT (inference) model and the GPT+DPR model.
Figure \ref{fig:AUC} shows the results.
In all settings, it can be recognized that the accuracy increases by narrowing down the questions to be answered only by those with high confidence scores, confirming the effectiveness of the confidence scores.
When comparing GPT (inference) and GPT+DPR, as shown in Table \ref{tab:EM}, the accuracy at an answer rate of 1.0 is higher for GPT+DPR in the case where a question completeness is 100\%, and equivalent in other cases. 
However, when an answer rate less than 0.8, GPT (inference) had a higher accuracy in all cases.
This difference is more obvious when the question completeness and the answer rate are low. 
For example, in the setting with a question completeness of 25\% and an answer rate of 0.1, the accuracy of GPT+DPR is around 0.5, while the accuracy of GPT (inference) is around 0.8, showing a significant difference.
Based on this, it can be said that GPT (inference) is more suitable for quick-answer quizzes.




\section{Conclusion}
In this study, we constructed two systems for answering buzzer quiz questions, which were not considered in previous research: GPT (inference) and GPT+DPR. 
Then we evaluated the accuracy for each question completeness. 
Furthermore, we investigated the relationship between confidence score and accuracy, and verified the validity of using the internal scores of the models as confidence scores.


\bibliography{reference}
\bibliographystyle{acl_natbib}



\end{document}
