Zhenan Fan
2025
DeTriever: Decoder-representation-based Retriever for Improving NL2SQL In-Context Learning
Raymond Li
|
Yuxi Feng
|
Zhenan Fan
|
Giuseppe Carenini
|
Weiwei Zhang
|
Mohammadreza Pourreza
|
Yong Zhang
Proceedings of the 31st International Conference on Computational Linguistics
While in-context Learning (ICL) has proven to be an effective technique to improve the performance of Large Language Models (LLMs) in a variety of complex tasks, notably in translating natural language questions into Structured Query Language (NL2SQL), the question of how to select the most beneficial demonstration examples remains an open research problem. While prior works often adapted off-the-shelf encoders to retrieve examples dynamically, an inherent discrepancy exists in the representational capacities between the external retrievers and the LLMs. Further, optimizing the selection of examples is a non-trivial task, since there are no straightforward methods to assess the relative benefits of examples without performing pairwise inference. To address these shortcomings, we propose Detriever, a novel demonstration retrieval framework that learns a weighted combination of LLM hidden states, where rich semantic information is encoded. To train the model, we propose a proxy score that estimates the relative benefits of examples based on the similarities between output queries. Experiments on two popular NL2SQL benchmarks demonstrate that our method significantly outperforms the state-of-the-art baselines for the NL2SQL tasks.
2024
Towards Human-aligned Evaluation for Linear Programming Word Problems
Linzi Xing
|
Xinglu Wang
|
Yuxi Feng
|
Zhenan Fan
|
Jing Xiong
|
Zhijiang Guo
|
Xiaojin Fu
|
Rindra Ramamonjison
|
Mahdi Mostajabdaveh
|
Xiongwei Han
|
Zirui Zhou
|
Yong Zhang
Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)
Math Word Problem (MWP) is a crucial NLP task aimed at providing solutions for given mathematical descriptions. A notable sub-category of MWP is the Linear Programming Word Problem (LPWP), which holds significant relevance in real-world decision-making and operations research. While the recent rise of generative large language models (LLMs) has brought more advanced solutions to LPWPs, existing evaluation methodologies for this task still diverge from human judgment and face challenges in recognizing mathematically equivalent answers. In this paper, we introduce a novel evaluation metric rooted in graph edit distance, featuring benefits such as permutation invariance and more accurate program equivalence identification. Human evaluations empirically validate the superior efficacy of our proposed metric when particularly assessing LLM-based solutions for LPWP.
Search
Fix data
Co-authors
- Yuxi Feng 2
- Yong Zhang 2
- Giuseppe Carenini 1
- Xiaojin Fu 1
- Zhijiang Guo 1
- show all...