Can LLMs Directly Retrieve Passages for Answering Questions from Qur’an?
Sohaila Eltanbouly, Salam Albatarni, Shaimaa Hassanein, Tamer Elsayed
Correct Metadata for
Abstract
The Holy Qur’an provides timeless guidance, addressing modern challenges and offering answers to many important questions. The Qur’an QA 2023 shared task introduced the Qur’anic Passage Retrieval (QPR) task, which involves retrieving relevant passages in response to MSA questions. In this work, we evaluate the ability of seven pre-trained large language models (LLMs) to retrieve relevant passages from the Qur’an in response to given questions, considering zero-shot and several few-shot scenarios. Our experiments show that the best model, Claude, significantly outperforms the state-of-the-art QPR model by 28 points on MAP and 38 points on MRR, exhibiting an impressive improvement of about 113% and 82%, respectively.- Anthology ID:
- 2025.arabicnlp-main.16
- Volume:
- Proceedings of The Third Arabic Natural Language Processing Conference
- Month:
- November
- Year:
- 2025
- Address:
- Suzhou, China
- Editors:
- Kareem Darwish, Ahmed Ali, Ibrahim Abu Farha, Samia Touileb, Imed Zitouni, Ahmed Abdelali, Sharefah Al-Ghamdi, Sakhar Alkhereyf, Wajdi Zaghouani, Salam Khalifa, Badr AlKhamissi, Rawan Almatham, Injy Hamed, Zaid Alyafeai, Areeb Alowisheq, Go Inoue, Khalil Mrini, Waad Alshammari
- Venue:
- ArabicNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 203–210
- Language:
- URL:
- https://aclanthology.org/2025.arabicnlp-main.16/
- DOI:
- Bibkey:
- Cite (ACL):
- Sohaila Eltanbouly, Salam Albatarni, Shaimaa Hassanein, and Tamer Elsayed. 2025. Can LLMs Directly Retrieve Passages for Answering Questions from Qur’an?. In Proceedings of The Third Arabic Natural Language Processing Conference, pages 203–210, Suzhou, China. Association for Computational Linguistics.
- Cite (Informal):
- Can LLMs Directly Retrieve Passages for Answering Questions from Qur’an? (Eltanbouly et al., ArabicNLP 2025)
- Copy Citation:
- PDF:
- https://aclanthology.org/2025.arabicnlp-main.16.pdf
Export citation
@inproceedings{eltanbouly-etal-2025-llms,
title = "Can {LLM}s Directly Retrieve Passages for Answering Questions from Qur{'}an?",
author = "Eltanbouly, Sohaila and
Albatarni, Salam and
Hassanein, Shaimaa and
Elsayed, Tamer",
editor = "Darwish, Kareem and
Ali, Ahmed and
Abu Farha, Ibrahim and
Touileb, Samia and
Zitouni, Imed and
Abdelali, Ahmed and
Al-Ghamdi, Sharefah and
Alkhereyf, Sakhar and
Zaghouani, Wajdi and
Khalifa, Salam and
AlKhamissi, Badr and
Almatham, Rawan and
Hamed, Injy and
Alyafeai, Zaid and
Alowisheq, Areeb and
Inoue, Go and
Mrini, Khalil and
Alshammari, Waad",
booktitle = "Proceedings of The Third Arabic Natural Language Processing Conference",
month = nov,
year = "2025",
address = "Suzhou, China",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2025.arabicnlp-main.16/",
pages = "203--210",
ISBN = "979-8-89176-352-4",
abstract = "The Holy Qur{'}an provides timeless guidance, addressing modern challenges and offering answers to many important questions. The Qur{'}an QA 2023 shared task introduced the Qur{'}anic Passage Retrieval (QPR) task, which involves retrieving relevant passages in response to MSA questions. In this work, we evaluate the ability of seven pre-trained large language models (LLMs) to retrieve relevant passages from the Qur{'}an in response to given questions, considering zero-shot and several few-shot scenarios. Our experiments show that the best model, Claude, significantly outperforms the state-of-the-art QPR model by 28 points on MAP and 38 points on MRR, exhibiting an impressive improvement of about 113{\%} and 82{\%}, respectively."
}<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="eltanbouly-etal-2025-llms">
<titleInfo>
<title>Can LLMs Directly Retrieve Passages for Answering Questions from Qur’an?</title>
</titleInfo>
<name type="personal">
<namePart type="given">Sohaila</namePart>
<namePart type="family">Eltanbouly</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Salam</namePart>
<namePart type="family">Albatarni</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Shaimaa</namePart>
<namePart type="family">Hassanein</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Tamer</namePart>
<namePart type="family">Elsayed</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<originInfo>
<dateIssued>2025-11</dateIssued>
</originInfo>
<typeOfResource>text</typeOfResource>
<relatedItem type="host">
<titleInfo>
<title>Proceedings of The Third Arabic Natural Language Processing Conference</title>
</titleInfo>
<name type="personal">
<namePart type="given">Kareem</namePart>
<namePart type="family">Darwish</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Ahmed</namePart>
<namePart type="family">Ali</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Ibrahim</namePart>
<namePart type="family">Abu Farha</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Samia</namePart>
<namePart type="family">Touileb</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Imed</namePart>
<namePart type="family">Zitouni</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Ahmed</namePart>
<namePart type="family">Abdelali</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Sharefah</namePart>
<namePart type="family">Al-Ghamdi</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Sakhar</namePart>
<namePart type="family">Alkhereyf</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Wajdi</namePart>
<namePart type="family">Zaghouani</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Salam</namePart>
<namePart type="family">Khalifa</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Badr</namePart>
<namePart type="family">AlKhamissi</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Rawan</namePart>
<namePart type="family">Almatham</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Injy</namePart>
<namePart type="family">Hamed</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Zaid</namePart>
<namePart type="family">Alyafeai</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Areeb</namePart>
<namePart type="family">Alowisheq</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Go</namePart>
<namePart type="family">Inoue</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Khalil</namePart>
<namePart type="family">Mrini</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Waad</namePart>
<namePart type="family">Alshammari</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<originInfo>
<publisher>Association for Computational Linguistics</publisher>
<place>
<placeTerm type="text">Suzhou, China</placeTerm>
</place>
</originInfo>
<genre authority="marcgt">conference publication</genre>
<identifier type="isbn">979-8-89176-352-4</identifier>
</relatedItem>
<abstract>The Holy Qur’an provides timeless guidance, addressing modern challenges and offering answers to many important questions. The Qur’an QA 2023 shared task introduced the Qur’anic Passage Retrieval (QPR) task, which involves retrieving relevant passages in response to MSA questions. In this work, we evaluate the ability of seven pre-trained large language models (LLMs) to retrieve relevant passages from the Qur’an in response to given questions, considering zero-shot and several few-shot scenarios. Our experiments show that the best model, Claude, significantly outperforms the state-of-the-art QPR model by 28 points on MAP and 38 points on MRR, exhibiting an impressive improvement of about 113% and 82%, respectively.</abstract>
<identifier type="citekey">eltanbouly-etal-2025-llms</identifier>
<location>
<url>https://aclanthology.org/2025.arabicnlp-main.16/</url>
</location>
<part>
<date>2025-11</date>
<extent unit="page">
<start>203</start>
<end>210</end>
</extent>
</part>
</mods>
</modsCollection>
%0 Conference Proceedings %T Can LLMs Directly Retrieve Passages for Answering Questions from Qur’an? %A Eltanbouly, Sohaila %A Albatarni, Salam %A Hassanein, Shaimaa %A Elsayed, Tamer %Y Darwish, Kareem %Y Ali, Ahmed %Y Abu Farha, Ibrahim %Y Touileb, Samia %Y Zitouni, Imed %Y Abdelali, Ahmed %Y Al-Ghamdi, Sharefah %Y Alkhereyf, Sakhar %Y Zaghouani, Wajdi %Y Khalifa, Salam %Y AlKhamissi, Badr %Y Almatham, Rawan %Y Hamed, Injy %Y Alyafeai, Zaid %Y Alowisheq, Areeb %Y Inoue, Go %Y Mrini, Khalil %Y Alshammari, Waad %S Proceedings of The Third Arabic Natural Language Processing Conference %D 2025 %8 November %I Association for Computational Linguistics %C Suzhou, China %@ 979-8-89176-352-4 %F eltanbouly-etal-2025-llms %X The Holy Qur’an provides timeless guidance, addressing modern challenges and offering answers to many important questions. The Qur’an QA 2023 shared task introduced the Qur’anic Passage Retrieval (QPR) task, which involves retrieving relevant passages in response to MSA questions. In this work, we evaluate the ability of seven pre-trained large language models (LLMs) to retrieve relevant passages from the Qur’an in response to given questions, considering zero-shot and several few-shot scenarios. Our experiments show that the best model, Claude, significantly outperforms the state-of-the-art QPR model by 28 points on MAP and 38 points on MRR, exhibiting an impressive improvement of about 113% and 82%, respectively. %U https://aclanthology.org/2025.arabicnlp-main.16/ %P 203-210
Markdown (Informal)
[Can LLMs Directly Retrieve Passages for Answering Questions from Qur’an?](https://aclanthology.org/2025.arabicnlp-main.16/) (Eltanbouly et al., ArabicNLP 2025)
- Can LLMs Directly Retrieve Passages for Answering Questions from Qur’an? (Eltanbouly et al., ArabicNLP 2025)
ACL
- Sohaila Eltanbouly, Salam Albatarni, Shaimaa Hassanein, and Tamer Elsayed. 2025. Can LLMs Directly Retrieve Passages for Answering Questions from Qur’an?. In Proceedings of The Third Arabic Natural Language Processing Conference, pages 203–210, Suzhou, China. Association for Computational Linguistics.