% This must be in the first 5 lines to tell arXiv to use pdfLaTeX, which is strongly recommended.
\pdfoutput=1
% In particular, the hyperref package requires pdfLaTeX in order to break URLs across lines.

\documentclass[11pt]{article}

% Remove the "review" option to generate the final version.
\usepackage[review]{acl}

% Standard package includes
\usepackage{times}
\usepackage{latexsym}
%\DeclareEmphSequence{\gtfamily\sffamily\bfseries}
% For proper rendering and hyphenation of words containing Latin characters (including in bib files)
\usepackage[T1]{fontenc}
% For Vietnamese characters
% \usepackage[T5]{fontenc}
% See https://www.latex-project.org/help/documentation/encguide.pdf for other character sets

% This assumes your files are encoded as UTF8
\usepackage[utf8]{inputenc}

% This is not strictly necessary, and may be commented out,
% but it will improve the layout of the manuscript,
% and will typically save some space.
\usepackage{microtype}

\newcommand{\yamada}[1]{\textcolor{magenta}{[Yamada: #1]}}
\newcommand{\bhline}{\noalign{\hrule height 1.2pt}}

% If the title and author information does not fit in the area allocated, uncomment the following
%
%\setlength\titlebox{<dim>}
%
% and set <dim> to something 5cm or larger.

\title{Instructions for *ACL Proceedings}

% Author information can be set in various styles:
% For several authors from the same institution:
% \author{Author 1 \and ... \and Author n \\
%         Address line \\ ... \\ Address line}
% if the names do not fit well on one line use
%         Author 1 \\ {\bf Author 2} \\ ... \\ {\bf Author n} \\
% For authors from different institutions:
% \author{Author 1 \\ Address line \\  ... \\ Address line
%         \And  ... \And
%         Author n \\ Address line \\ ... \\ Address line}
% To start a seperate ``row'' of authors use \AND, as in
% \author{Author 1 \\ Address line \\  ... \\ Address line
%         \AND
%         Author 2 \\ Address line \\ ... \\ Address line \And
%         Author 3 \\ Address line \\ ... \\ Address line}

\author{First Author \\
  Affiliation / Address line 1 \\
  Affiliation / Address line 2 \\
  Affiliation / Address line 3 \\
  \texttt{email@domain} \\\And
  Second Author \\
  Affiliation / Address line 1 \\
  Affiliation / Address line 2 \\
  Affiliation / Address line 3 \\
  \texttt{email@domain} \\}

\begin{document}
\maketitle
\begin{abstract}
A “quick-buzz quiz” is a genre of competitive quiz in which participants can answer while the question is still being read, emphasizing the speed of answering. Due to this nature, it often involves answering based on incomplete sentences.
In this study, we work on constructing a quick-buzz quiz answering system that outputs appropriate answers to incomplete questions. Specifically, we develop and evaluate two systems: one that directly generates answers after the question sentence using GPT, and another that first completes the question sentence using GPT and then generates answers through a Retriever-Reader approach using DPR.

\end{abstract}

\section{Introduction}
Competitive quizzes are widely researched as open-domain question answering tasks, as they do not limit the range of knowledge to be tested. However, the main datasets used for research in this area, such as Natural Questions \cite{kwiatkowski-etal-2019-natural} and TriviaQA \cite{joshi-etal-2017-triviaqa}, feature complete question sentences, and systems built using these datasets \cite{karpukhin-etal-2020-dense, yamada-etal-2021-efficient, izacard-grave-2021-leveraging} are not designed to answer incomplete questions.





In addition, in quick-buzz quizzes, it is important not only to provide an answer but also to consider the likelihood of the predicted answer to the given question at the current point and judge whether to actually answer it.
For example, considering the question shown in Table \ref{tab:example}, if only "In ancient Egypt" is read, there are many things that existed in ancient Egypt other than the actual answer "Sirius," so the predicted answer at that point is likely to be incorrect. On the other hand, if the question is read until the end, the answer is almost uniquely determined. Therefore, by calculating the confidence level that each answer candidate is the actual correct answer at each point in the question, a higher-performing quick-buzz quiz answering system can be constructed.
This study aims to first construct a quick-buzz quiz answering system that can output appropriate answers to incomplete questions, and then calculate the confidence levels for each model and investigate the relationship between confidence levels and accuracy. Specifically, two systems will be built: one uses GPT~\cite{radford-2018-improving} to generate answers directly after the question, while the other uses a Retriever-Reader approach with Dense Passage Retrieval (DPR)\cite{karpukhin-etal-2020-dense} after question completion with GPT. Confidence levels will be calculated using token output probabilities for the former system, and the model's score used in generating the output for the latter system. The confidence levels shown in Table\ref{tab:example} are based on the former system's score, and it can be confirmed that the confidence levels are functioning appropriately in the quick-buzz quiz system, where low confidence levels correspond to incorrect answers and high confidence levels correspond to correct answers.
In the former system, the token output probabilities are used to calculate the confidence score during answer generation, while in the latter system, the scores used in the model's output are used to calculate the confidence score. The confidence scores listed in Table \ref{tab:example} are based on the scores from the former system, and if the confidence score is low, an incorrect answer is output, whereas if the confidence score is high, a correct answer is output. This confirms that the confidence score functions appropriately in the quick-buzz quiz system.



\section{Quiz answering systems}
\label{sec:DPRandGPT}
In this section, we first introduce previous research on competitive quizzes, and then propose a quick-buzz quiz answer system that can handle incomplete questions, as well as a method for calculating confidence based on model output.

\subsection{Open-domain question answering system}
\label{sec:OpenDomain}
Competitive quizzes are treated as open-domain question answering tasks, with two main approaches for answer generation: a single model that directly generates answers, and a retriever-reader approach that treats it as a composite task of searching for relevant documents and extracting answers.
As an example of a single-model answer, GPT, a pre-trained language model trained on a large text corpus, based on the Transformer~\cite{vaswani-2017-attention} decoder, is often used. GPT can be used for language generation-related tasks, such as generating text following an input text, and in the case of question answering, it can generate answers by inputting a format that infers only answers to the question.
Furthermore, in GPT, fine-tuning using downstream task data is expected to achieve higher performance, so fine-tuning is often performed in the construction of question answering models.
Retriever-Reader approach based on DPR using Retriever is the representative example for answering composite tasks. DPR has a dual-encoder structure that uses different BERT~\cite{devlin-2019-bert} for each document and the query, and obtains embedding representations of the query and each document by inserting a special token \verb|[CLS]| at the beginning of each document when inputting to BERT. The document selection is then performed based on the semantic similarity calculated by the inner product of these representations \cite{karpukhin-etal-2020-dense}. In the Reader, the model performs two tasks: predicting the related documents including the correct answer using BERT, and extracting the answer span within the selected document. In the related document prediction task, the model predicts the documents that may contain the answer based on the position of the \verb|[CLS]| token. Then, for the selected document, the model extracts the answer span by predicting the beginning and ending tokens of the answer within the document and outputs them.


\subsection{Quick-buzz quiz answering system}
The systems in the previous section are surely effective for answering complete sentences, but its effectiveness is unknown when answering quick-buzz quizzes that require answering incomplete questions. Generally, when only part of a question is given, there may be various possible answers and necessary information to answer may not be provided, making the nature of the problem significantly different from when a complete sentence is given. Therefore, it is considered that specialized methods are necessary for quick-buzz quiz tasks.
In this study, we constructed two systems: \emph{GPT (inference)} and \emph{GPT+DPR}, which rely solely on inference using the large-scale language model GPT, and perform question completion using GPT followed by the Retriever-Reader approach using DPR, respectively.
In GPT (inference), the input format is set to "question + / the answer is " to allow inference on the part after the parenthesis and output the contents inside the parenthesis as the answer. By inserting "/" between the question and "the answer is", it recognizes the boundary of the question and prevents incorrect sentence completion.
In GPT+DPR system, incomplete sentences are input into GPT for completion, and the completed sentences are then used as input to the Retriever-Reader model based on DPR for answering.




\subsection{Confidence scores in each models}

In this study, we calculate a confidence score for each model's answer by using the internal score that is calculated to generate the output for each model. The confidence score is a value used as an indicator to judge whether the model's answer is correct or not, and it is necessary for the confidence score to have the property that the higher the score, the higher the accuracy of the answer.
On the other hand, for the GPT+DPR, three internal scores can be used as confidence measures: the \emph{document relevance score}, the \emph{extraction score}, and their average, the \emph{average score}. In the Reader, a linear layer is trained on the \verb|[CLS]| token of each document to score them, and the document with the highest score is selected, which is the document relevance score. Then, the selected document is converted into embedded representations for each token and fed into another linear layer to calculate the start and end scores. The token sequence from the token with the highest start score to the token with the highest end score is selected as the answer. The sum of the start and end scores is defined as the extraction score.

\section{Experiments}

We conducted experiments to verify the accuracy of the proposed quick response quiz answering system and investigate the effectiveness of the confidence levels of each model. Note that the question completeness percentage when the problem statement is truncated at the top x\% using character-based processing is denoted as "question completeness x\%". In the accuracy verification experiment, we applied the GPT (inference) model and GPT+DPR model described in Section \ref{sec:hayaoshi} to problems with question completeness percentages of 25\%, 50\%, 75\%, and 100\%, and verified their accuracy. In the confidence level effectiveness investigation experiment, we evaluated the confidence level by investigating the relationship between the confidence level and the correct answer rate for each question completeness percentage using the confidence levels of each model described in Section \ref{sec:kakushindo}.

\paragraph{Datasets}
In this paper, we used data from the Second AI King Official Distribution Dataset (hereinafter referred to as AI King)\footnote{\url{https://sites.google.com/view/project-aio/dataset}} and data crawled from the quiz app "Minna de Hayaoshi Quiz" (hereinafter referred to as "Minhaya")\footnote{\url{https://livequiz.work/minhaya1/}}. The number of data and average number of characters are described in Table \ref{tab:data}. Note that since DPR training requires positive and negative example documents in addition to the question and answer, we only used the AI King data for DPR training and the added Minhaya data for GPT training.

\paragraph{Learning Settings}
We compared two models: GPT (inference) and GPT+DPR. For GPT, we used the model released by rinna on Hugging Face \cite{wolf-2020-transformers} \footnote{\url{https://huggingface.co/rinna/japanese-gpt-1b}}. For DPR, we used the baseline model released in the 2nd AI King competition \footnote{\url{https://github.com/cl-tohoku/AIO2\_DPR\_baseline}}. GPT (inference) was fine-tuned on the training data formatted as "question + '/the answer is "○○"'. For GPT+DPR, we fine-tuned GPT only on the question part of the training data. In both cases, we trained the models for 5 epochs. For DPR, we used a pre-trained BERT model \footnote{\url{https://huggingface.co/cl-tohoku/bert-large-japanese}} as the base model for both Retriever and Reader. Retriever was trained with a batch size of 128, a learning rate of 1e-5, and 5 epochs, while Reader was trained with a batch size of 8, a learning rate of 2e-5, and 3 epochs.

\paragraph{Comparison method}
In the accuracy verification experiment, two models, GPT (inference) and GPT+DPR, were compared. In the confidence level effectiveness investigation experiment, the generated score was used for GPT (inference), while for GPT+DPR, three scores, the related document score, extraction score, and average score, were used.

\paragraph{Evaluation}
In the accuracy evaluation, the correctness of the model output was evaluated by exact string matching. However, to prevent incorrect answers due to variations in notation, six symbols "(~)~[~]~・~=" were mechanically removed from both the model output and the answer before comparing the strings.

In the confidence effectiveness experiment, a correct answer rate-answer rate curve was created, and the effectiveness of confidence was investigated by calculating the area under the curve (AUC). Here, the answer rate refers to the percentage of cases in which the system actually outputs an answer. If only answers whose confidence exceeds a threshold alpha are considered as correct, the answer rate can be controlled by varying this threshold alpha. On the other hand, the correct answer rate refers to the percentage of cases in which the system output an answer that was correct among all answers. If alpha is less than zero, this is equivalent to the overall accuracy of the system, and as alpha increases, only answers with high confidence are evaluated, so it is expected that the correct answer rate will increase.


\subsection{Results:Accuracy verification}
Table \ref{tab:EM} shows the accuracy of GPT (inference) and GPT+DPR for each level of question completeness. As the completeness of the questions decreased, the accuracy of both models also decreased, but the decline was not proportional to the completeness level. The drop-off was relatively gradual between 100\% and 75\%. This is thought to be because important words that determine the answer appear frequently in the first half to middle of the quiz question, while there are fewer cases where words with a lot of information appear in the second half. When comparing the scores of the two models, GPT+DPR showed higher performance when the question completeness was 100\%, but no difference in performance was observed between the two models when the question was incomplete.








\subsection{Results:Effectiveness of confidence scores}

The results of the effectiveness evaluation experiment for confidence levels are shown in Table \ref{tab:AUC}. Among the three confidence levels used for GPT+DPR, the one using the related document score achieved the highest AUC. Furthermore, among all the results, the generation score of GPT (inference) resulted in the highest AUC.
Next, we compared the PR curves of the GPT (inference) model and the GPT+DPR model using the related document score with the highest AUC in GPT+DPR. The results are shown in Figure \ref{fig:my_label}. In all settings, it was confirmed that the accuracy increased by narrowing down the questions that were answered only with high confidence, confirming the effectiveness of confidence. Comparing the GPT (inference) model and the GPT+DPR model, the accuracy rate, which is the correct answer rate at a response rate of 1.0 as shown in Table \ref{tab:EM}, is higher for GPT+DPR when the question text completeness is 100\%, and they are roughly equal in other cases, but for response rates of less than 0.8, both models had higher accuracy rates with GPT (inference) having a more significant difference. This difference was particularly pronounced when the question text completeness or response rate was low, and for example, in settings with a question text completeness of 25\% and a response rate of 0.1, the accuracy rate of GPT+DPR was about 0.5, while that of GPT (inference) was about 0.8, showing a large difference. Therefore, it can be said that the GPT (inference) model is more suitable for rapid-fire quizzes.
The output examples of applying GPT~(inference) to problems with 25\% question completeness are shown in Table \ref{tab:correct}.
For questions 1 and 2, even though only the first 25\% of the question was given, the confidence score exceeded 0.9, and the correct answers were actually output.
On the other hand, for question 3, although the confidence score was high, an incorrect answer was actually output, probably because the model did not take into account the so-called "desu ga problem" when making predictions.
In question 4, since the answer candidate could not be narrowed down to one with the first 25\%, the confidence score is considered to have been low, and the correct answer, "Aho-dori," could not be output, and an incorrect answer, "Namakemono," was output instead.







\section{Conclusion}
In this study, we constructed two types of systems, GPT (inference) and GPT+DPR, to answer rapid-fire quiz questions that were not assumed in previous research, and verified the accuracy rate for each question completeness. In addition, we investigated the relationship between confidence and accuracy rates, and validated the validity of using the model's internal score as confidence.
As a result of the accuracy rate verification, although DPR showed the highest score for complete sentences, there was no difference in the scores of both models for incomplete sentences.
In the investigation of the effectiveness of confidence, it was confirmed that the confidence score based on the GPT (inference) model showed high effectiveness even for questions with low completeness.
In the future, we want to construct a system that determines the timing of answers by setting a confidence threshold using the confidence targeted in this study, and a real-time rapid-fire quiz answering model.



% Entries for the entire Anthology, followed by custom entries
\bibliography{anthology,custom}
\bibliographystyle{acl_natbib}



\end{document}
