%0 Conference Proceedings %T What Can a Generative Language Model Answer About a Passage? %A Summers-Stay, Douglas %A Bonial, Claire %A Voss, Clare %Y Fisch, Adam %Y Talmor, Alon %Y Chen, Danqi %Y Choi, Eunsol %Y Seo, Minjoon %Y Lewis, Patrick %Y Jia, Robin %Y Min, Sewon %S Proceedings of the 3rd Workshop on Machine Reading for Question Answering %D 2021 %8 November %I Association for Computational Linguistics %C Punta Cana, Dominican Republic %F summers-stay-etal-2021-generative %X Generative language models trained on large, diverse corpora can answer questions about a passage by generating the most likely continuation of the passage followed by a question/answer pair. However, accuracy rates vary depending on the type of question asked. In this paper we keep the passage fixed, and test with a wide variety of question types, exploring the strengths and weaknesses of the GPT-3 language model. We provide the passage and test questions as a challenge set for other language models. %R 10.18653/v1/2021.mrqa-1.7 %U https://aclanthology.org/2021.mrqa-1.7 %U https://doi.org/10.18653/v1/2021.mrqa-1.7 %P 73-81