Xingyu Cui


2024

pdf bib
Prompt Engineering 101 Prompt Engineering Guidelines from a Linguistic Perspective
Wenjuan Han | Xiang Wei | Xingyu Cui | Ning Cheng | Guangyuan Jiang | Weinan Qian | Chi Zhang
Proceedings of the 23rd Chinese National Conference on Computational Linguistics (Volume 1: Main Conference)

“Deploying tuning-free prompting is challenging in engineering practice: it not only requiresusers to engage in cumbersome trials and errors but is also extremely time-consuming,as even a slight change in wording and phrasing could have a huge impact on the finalperformance. To further investigate the impact of different prompts, in this work, weperform a systematic inspection of four factors in linguistics involved in prompt engineering:syntax, semantics, lexicon, and pragmatics. The empirical results quantify the sensitivityof the output to small textual perturbations in four linguistic factors of prompts. Basedon the analysis of these four factors, we present a series of design guidelines to helphuman users write effective prompts. Human evaluation on amateurs shows that usingthe proposed guidelines helps humans produce prompts with significant gains in zero-shotperformance in Pre-trained Language Models (PLMs) and hence validates the utility ofthe guidelines.”

pdf bib
CollabKG: A Learnable Human-Machine-Cooperative Information Extraction Toolkit for (Event) Knowledge Graph Construction
Xiang Wei | Yufeng Chen | Ning Cheng | Xingyu Cui | Jinan Xu | Wenjuan Han
Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)

In order to construct or extend entity-centric and event-centric knowledge graphs (KG and EKG), the information extraction (IE) annotation toolkit is essential. However, existing IE toolkits have several non-trivial problems, such as not supporting multi-tasks, and not supporting automatic updates. In this work, we present CollabKG, a learnable human-machine-cooperative IE toolkit for KG and EKG construction. Specifically, for the multi-task issue, CollabKG unifies different IE subtasks, including named entity recognition (NER), entity-relation triple extraction (RE), and event extraction (EE), and supports both KG and EKG. Then, combining advanced prompting-based IE technology, the human-machine-cooperation mechanism with Large Language Models (LLMs) as the assistant machine is presented which can provide a lower cost as well as a higher performance. Lastly, owing to the two-way interaction between the human and machine, CollabKG with learning ability allows self-renewal. Besides, CollabKG has several appealing features (e.g., customization, training-free, and label propagation) that make the system powerful and high-productivity. We holistically compare our toolkit with other existing tools on these features. Human evaluation quantitatively illustrates that CollabKG significantly improves annotation quality, efficiency, and stability simultaneously.