Query-OPT: Optimizing Inference of Large Language Models via Multi-Query Instructions in Meeting Summarization

Md Tahmid Rahman Laskar, Elena Khasanova, Xue-Yong Fu, Cheng Chen, Shashi Bhushan Tn


Abstract
This work focuses on the task of query-based meeting summarization in which the summary of a context (meeting transcript) is generated in response to a specific query. When using Large Language Models (LLMs) for this task, a new call to the LLM inference endpoint/API is required for each new query even if the context stays the same. However, repeated calls to the LLM inference endpoints would significantly increase the costs of using them in production, making LLMs impractical for many real-world use cases. To address this problem, in this paper, we investigate whether combining the queries for the same input context in a single prompt to minimize repeated calls can be successfully used in meeting summarization. In this regard, we conduct extensive experiments by comparing the performance of various popular LLMs: GPT-4, Gemini, Claude-3, LLaMA2, Mistral, Phi-3, and Qwen-2 in single-query and multi-query settings. We observe that the capability to reliably generate the response in the expected format is usually limited to closedsource LLMs, with most open-source LLMs lagging behind (except Mistral). We conclude that multi-query prompting could be useful to optimize the inference costs by significantly reducing calls to the inference endpoints/APIs for the task of meeting summarization.
Anthology ID:
2024.emnlp-industry.86
Volume:
Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track
Month:
November
Year:
2024
Address:
Miami, Florida, US
Editors:
Franck Dernoncourt, Daniel Preoţiuc-Pietro, Anastasia Shimorina
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1140–1151
Language:
URL:
https://aclanthology.org/2024.emnlp-industry.86
DOI:
10.18653/v1/2024.emnlp-industry.86
Bibkey:
Cite (ACL):
Md Tahmid Rahman Laskar, Elena Khasanova, Xue-Yong Fu, Cheng Chen, and Shashi Bhushan Tn. 2024. Query-OPT: Optimizing Inference of Large Language Models via Multi-Query Instructions in Meeting Summarization. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 1140–1151, Miami, Florida, US. Association for Computational Linguistics.
Cite (Informal):
Query-OPT: Optimizing Inference of Large Language Models via Multi-Query Instructions in Meeting Summarization (Laskar et al., EMNLP 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.emnlp-industry.86.pdf