Tracking progress on climate change adaptation requires systematic and robust annotation of adaptation policies, but manual annotation has become infeasible due to the rapidly growing volume of unstructured documents. Pre-trained language models (LMs) have emerged as promising tools for automating and scaling annotation. However, relatively little is known about their accuracy and quality on complex, free-text annotation tasks. Meanwhile, large LMs (LLMs) are increasingly deployed for large scale text analyses despite their considerable environmental footprint and limited exploration of more sustainable alternatives. By benchmarking model accuracy against human inter-annotator agreement, reviewing perceived annotation quality, and estimating energy and water consumption, we show that fine-tuning small LMs (SLMs) provides a substantially more sustainable alternative to LLMs with only marginal reductions in accuracy and improved perceived annotation quality. Furthermore, we demonstrate the importance of evaluating complex annotation tasks beyond accuracy against human gold standard annotation as ambiguity and human error can obscure meaningful differences in model performance. By reducing the costs of high-quality policy annotation, our approach enables the scalable knowledge extraction needed for tracking progress on adaptation, living evidence syntheses, and evidence-informed policy evaluation.

The voluminous, highly unstructured, and intersectoral nature of climate policy data resulted in increased calls for automated methods to retrieve information relevant to climate change adaptation. Collecting such information is crucial to establish a large-scale evidence base to monitor and evaluate current adaptation practices. Using a novel, hand-labelled dataset, we explored the potential of state-of-the-art Natural Language Processing methods and compared the performance of various Transformer-based solutions to classify text based on adaptation-relevance in both zero-shot and fine-tuned settings. We find that fine-tuned, encoder-only models, particularly those pre-trained on data from a related domain, are best suited to the task, outscoring zero-shot and rule-based approaches. Furthermore, our results show that text granularity played a crucial role in performance, with shorter text splits leading to decreased performance. Finally, we find that excluding records with below-moderate annotator confidence enhances model performance. These findings reveal key methodological considerations for automating and upscaling text classification in the climate change (adaptation) policy domain.