Lakshmi Mandal
2024
Sequential API Function Calling Using GraphQL Schema
Avirup Saha
|
Lakshmi Mandal
|
Balaji Ganesan
|
Sambit Ghosh
|
Renuka Sindhgatta
|
Carlos Eberhardt
|
Dan Debrunner
|
Sameep Mehta
Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing
Function calling using Large Language Models (LLMs) is an active research area that aims to empower LLMs with the ability to execute APIs to perform real-world tasks. However, sequential function calling using LLMs with interdependence between functions is still under-explored. To this end, we introduce GraphQLRestBench, a dataset consisting of natural language utterances paired with function call sequences representing real-world REST API calls with variable mapping between functions. In order to represent the response structure of the functions in the LLM prompt, we use the GraphQL schema of the REST APIs. We also introduce a custom evaluation framework for our dataset consisting of four specially designed metrics. We evaluate various open-source LLMs on our dataset using few-shot Chain-of-Thought and ReAct prompting to establish a reasonable baseline.
Search
Co-authors
- Avirup Saha 1
- Balaji Ganesan 1
- Sambit Ghosh 1
- Renuka Sindhgatta 1
- Carlos Eberhardt 1
- show all...