% This must be in the first 5 lines to tell arXiv to use pdfLaTeX, which is strongly recommended.
\pdfoutput=1
% In particular, the hyperref package requires pdfLaTeX in order to break URLs across lines.

\documentclass[11pt]{article}

% Remove the "review" option to generate the final version.
\usepackage[review]{acl}

% Standard package includes
\usepackage{times}
\usepackage{latexsym}
%\DeclareEmphSequence{\gtfamily\sffamily\bfseries}
% For proper rendering and hyphenation of words containing Latin characters (including in bib files)
\usepackage[T1]{fontenc}
% For Vietnamese characters
% \usepackage[T5]{fontenc}
% See https://www.latex-project.org/help/documentation/encguide.pdf for other character sets

% This assumes your files are encoded as UTF8
\usepackage[utf8]{inputenc}

% This is not strictly necessary, and may be commented out,
% but it will improve the layout of the manuscript,
% and will typically save some space.
\usepackage{microtype}

\newcommand{\yamada}[1]{\textcolor{magenta}{[Yamada: #1]}}
\newcommand{\bhline}{\noalign{\hrule height 1.2pt}}

% If the title and author information does not fit in the area allocated, uncomment the following
%
%\setlength\titlebox{<dim>}
%
% and set <dim> to something 5cm or larger.

\title{Instructions for *ACL Proceedings}

% Author information can be set in various styles:
% For several authors from the same institution:
% \author{Author 1 \and ... \and Author n \\
%         Address line \\ ... \\ Address line}
% if the names do not fit well on one line use
%         Author 1 \\ {\bf Author 2} \\ ... \\ {\bf Author n} \\
% For authors from different institutions:
% \author{Author 1 \\ Address line \\  ... \\ Address line
%         \And  ... \And
%         Author n \\ Address line \\ ... \\ Address line}
% To start a seperate ``row'' of authors use \AND, as in
% \author{Author 1 \\ Address line \\  ... \\ Address line
%         \AND
%         Author 2 \\ Address line \\ ... \\ Address line \And
%         Author 3 \\ Address line \\ ... \\ Address line}

\author{First Author \\
  Affiliation / Address line 1 \\
  Affiliation / Address line 2 \\
  Affiliation / Address line 3 \\
  \texttt{email@domain} \\\And
  Second Author \\
  Affiliation / Address line 1 \\
  Affiliation / Address line 2 \\
  Affiliation / Address line 3 \\
  \texttt{email@domain} \\}

\begin{document}
\maketitle
\begin{abstract}
In this study, we aim to develop a buzzer quiz answering system that can output appropriate answers to incomplete quiz questions.
Specifically, we will build two systems: one that directly generates answers after the question using GPT, and the other uses GPT to complete the question and then generates answers by the retriever-reader approach with DPR.
In these buzzer quiz answering systems, it is crucial to assess the plausibility of the answer at each point in time. We also propose a method for calculating confidence score, which serve as an indicator for this purpose. As a result, by answering only questions with high confidence score, we achieved a high accuracy rate even when the overall accuracy was low.


\end{abstract}

\section{Introduction}
A buzzer quiz is a genre of quiz in which players answer by buzzing as soon as they can predict an answer, while the questioner reading aloud the quiz text. 
The importance of buzzer quizzes is rapidness of buzzing, so players normally answer to incomplete questions.
Buzzer quizzes have been widely studied as open-domain question answering tasks because they do not limit the scope of knowledge.
However, major datasets of open-domain question answering tasks like Natural Questions \cite{kwiatkowski-etal-2019-natural} and TriviaQA \cite{joshi-etal-2017-triviaqa} consist of complete question sentences. 
Consequently, systems built using these datasets \cite{karpukhin-etal-2020-dense,yamada-etal-2021-efficient,izacard-grave-2021-leveraging} are not designed to answer incomplete questions.

Furthermore, in buzzer quizzes, it is certainly crucial to providing an correct answer, but it is also essential to consider the plausibility of the predicted answer based on the given question at that moment and decide whether to actually respond or not.
For example, consider a question as shown in Table \ref{tab:example} when it hasn't been read up to ``200-hit.''
At this point, as there are other baseball players who hold records comparable to Pete Rose, it is difficult to narrow down the answer to a single candidate.
This makes the predicted answer at that moment more likely to be incorrect, so we shouldn't answer at this time.
On the other hand, if the question is read further, the predicted answer converges to the correct answer, ``Ichiro Suzuki.''

Therefore, it is conceivable that a more effective Buzzer Quiz answering system can be constructed by calculating the confidence score for the candidate answers at each point in the question text, indicating the likelihood of them being the correct answer.
In addition, the ability to respond to Buzzer Quizzes, answering to incomplete questions, could contribute to replicating the human capacity to smoothly generate responses in a conversation by sequentially predicting the content of the dialogue.

\begin{table}[t!]
\centering
\small

\label{tab:example}
\begin{tabular}{@{\ }l@{\ }} \bhline
\rule{0pt}{2.5ex}
\textbf{75\% completeness}: Pete Rose and this player are \\tied with
ten 200-hit seasons each. This Japanese \\outfielder played most
of his career with the Mar\textcolor[gray]{0.7}{iners,} \\\textcolor[gray]{0.7}{and currently plays for the Marlins.}\\
\rule{0pt}{2.5ex}
confidence score: 0.991 A: Ichiro Suzuki \ \ \emph{\color{red}correct} \\\hline

\rule{0pt}{2.5ex}
\textbf{25\% completeness}: Pete Rose and this player are \\tied with
ten 2\textcolor[gray]{0.7}{00-hit seasons each. This Japanese}\\
\textcolor[gray]{0.7}{outfielder played most
of his career with the Mariners,}\\
\textcolor[gray]{0.7}{and currently plays for the Marlins.}\\
\rule{0pt}{2.5ex}
confidence score: 0.125  A: Ty Cobb \ \ \emph{\color{blue}incorrect} \\\hline

\rule{0pt}{2.5ex}

\rule{0pt}{2.5ex}

\end{tabular}
\caption{An output example}
\end{table}


In this study, we first construct a Buzzer Quiz answering system that produces appropriate answers for incomplete questions, and we propose the method of calculating the confidence score for each model.
Specifically, we construct two systems: one that generates answers directly following the question text using GPT \cite{radford-2018-improving}, and another that generates answers through a Retriever-Reader approach using Dense Passage Retrieval (DPR) \cite{karpukhin-etal-2020-dense} after completing the question text with GPT. 
For the first system, we calculate confidence score using token output probabilities during answer generation, while for the second system, we use scores employed in the  output of the model. 
The confidence scores shown in Table \ref{tab:example} represent the scores from the first system. 
When the confidence score is low, there are various possible answers because the completeness of the question is low, whereas more information of correct answer is provided, the confidence score become higher. 
This shows that the confidence score is functioning appropriately in the buzzer quiz system.




\section{Quiz answering systems}
\label{sec:DPRandGPT}
In this section, we first introduce related research on quiz. Subsequently, we propose a Buzzer Quiz answering system that answers incomplete question sentences and a method for calculating confidence score based on the model's output.

\subsection{Open-domain question answering system}
\label{sec:OpenDomain}
Quizzes are studied as open-domain question-answering task, and there are two mainstream approaches to answering models: a single model that directly generates answers and a retriever-reader approach that answers by combining two tasks, searching for relevant documents and extracting answer.
GPT \cite{radford-2018-improving} is a representative example of a single model for generating answers.
GPT is a pre-trained language model based on the Transformer decoder ~\cite{vaswani-2017-attention} that uses a large-scale text corpus to learn to predict the subsequent word given a sequence of words

A representative example of an answering model for compound tasks is the retriever-reader model based on DPR. 
Retriever employs a dual-encoder structure using different BERT ~\cite{devlin-2019-bert} models for the question text and each document. When inputting into BERT, a special token \verb|[CLS]| is inserted at the beginning of the document, and the embedding representations for the question text and each document are obtained.
And then this method selects documents based on semantic similarity calculated by the inner product of the obtained representations\cite{karpukhin-etal-2020-dense}.
In the reader, BERT predicts the relevant documents containing the correct answer and extracts the answer portion within the document. 
For the related document prediction, it predicts the document likely to contain the answer based on the \verb|[CLS]| token position. Then, for the chosen document, it performs the answer portion extraction prediction, determining the starting and ending points of the token sequence that forms the answer and outputs it.



\subsection{Buzzer quiz answering system}
While the effectiveness of the systems mentioned in the previous section has been confirmed for complete sentences, their effectiveness in buzzer quizzes, where normally requires to answer to incomplete sentences, remains unclear.
Generally, when only a part of a question is given, the nature of the problem differs significantly from that of a complete question, because there may be multiple possible answers, or the necessary information to determine the answer might not be given.
Therefore, it is believed that a specific method for buzzer quiz tasks is necessary.

In this study, we construct two systems: one that relies solely on inference using the large-scale language model GPT, referred to as the \textbf{GPT (Inference)} system, and another that utilizes GPT for question completion and employs the Retriever-Reader approach with DPR, called the \textbf{GPT+DPR} system.
In the GPT (Inference) system, the input format is designed as ``question text + '/the answer is` '', which prompts the model to generate the answer within the quotation marks. And then the content inside the quotation marks is used as the predicted answer.
Inserting a '/' between the question text and ``the answer is` '' is aiming to made the model to recognize the boundary of the question text, which prevents incomplete question completion.
In the GPT+DPR system, the incomplete sentence is input into GPT to complete the question text, and the resulting complete sentence is then used as input for the DPR-based Retriever-Reader model to generate the answer. This approach combines the strengths of GPT for text completion and the effectiveness of DPR for retrieving and extracting answers from relevant documents.

\subsection{Confidence scores in each models}
In this study, we calculate the confidence score for predicted answers using the internal scores that are utilized by each model when generating their outputs.
Confidence score is a value used as an indicator for determining whether a predicted answer is correct or not, and it is necessary for the confidence score to have a property such that the larger the confidence score, the higher the accuracy rate.

In the GPT (inference) model, we use the generation probability of the first token in the predicted answer (referred to as the \textbf{generation score}) as the confidence score.
In GPT, during sentence generation, given the sentence composed of $n$ tokens, the model outputs the token with the highest generation score from the vocabulary as the $n+1$th token.
In buzzer quiz, the answer is often composed of a small number of tokens. 
So the first token largely determines the direction of the answer. Therefore, only the generation score of the first token was adopted as the confidence score.
On the other hand, for the GPT+DPR model, we consider three internal scores that can be used as confidence scores: the \textbf{related document score} and the \textbf{extraction score} calculated by the Reader, as well as their arithmetic mean, the \textbf{average score}.
In the Reader, each [CLS] token of the document is scored using a learned linear layer, and the document with the highest score is selected. This score is the related document score.
Furthermore, after selecting a document, the document is converted into token-level embedding representations. Then, we calculate a span score that is composed by a start score and a end score. The answer is the sequence of tokens from the token with the highest start score to the token with the highest end score. The extraction score is the sum of these start and end scores.






\section{Experiments}



\paragraph{Datasets}


\paragraph{Learning Settings}


\paragraph{Comparison method}


\paragraph{Evaluation}



\subsection{Results:Accuracy verification}









\subsection{Results:Effectiveness of confidence scores}








\section{Conclusion}




% Entries for the entire Anthology, followed by custom entries
\bibliography{anthology,custom}
\bibliographystyle{acl_natbib}



\end{document}
