Samuel Forster
Author directory2026
FAST-Eval and FAST-LoRA: Uncertainty Evaluation Framework and Low-Rank Adaptation for Finding Attributes with Supporting Text
Samuel Forster | Inmaculada Arnedillo-Sánchez
The Proceedings of the Third Workshop on Uncertainty-Aware NLP
Samuel Forster | Inmaculada Arnedillo-Sánchez
The Proceedings of the Third Workshop on Uncertainty-Aware NLP
Many domains require reasoning over unstructured text to identify attributes of interest that are not stated in a standardised form. Useful judgements on the presence of these attributes should also express calibrated confidence statements and cite supporting text as evidence. We define this task as Finding Attributes with Supporting Text (FAST). We introduce FAST-Eval, an uncertainty evaluation framework for confidence calibration, attribute identification and evidential grounding in FAST, together with a dataset of 1,000 human-annotated (document, attribute) pairs validated against an independent recruitment expert (Spearman ρs = 0.853, binary κ = 0.715). Taking employability skills in CVs as a case study, we use FAST-Eval to test whether FAST-LoRA, a Low-Rank Adaptation (LoRA) adapter applied to Qwen3-8B, improves performance in these three dimensions. Trained on 500 pairs, FAST-LoRA outperforms its untuned Qwen3-8B counterpart and GPT-5 on unseen skill identification (F1 0.877 vs 0.736 and 0.766) and calibration, while matching GPT-5 on citation fidelity. Both untuned models assign the highest confidence level to about half their judgements; FAST-LoRA instead distributes confidence across the scale and makes fewer unsupported positive predictions. These results suggest that modest domain adaptation is a more effective strategy than scaling model parameters for FAST.