A Scalable Pipeline for Novelty Detection in Skill Extraction Using Large Language Models

Gian Seifert, Simon Clematide


Abstract
The rapid evolution of the labor market requires skill ontologies to be continuously updated, but manually identifying emerging skills in job advertisements is highly labor-intensive. This paper presents a scalable, multi-stage pipeline for automated novelty detection in skill extraction. The system combines Large Language Models (LLMs) for candidate generation, a re-matching and threshold-based filtering module (“Turbo”), that compares candidates against the existing ontology, and a two-step aggregation process that merges string-based and embedding-based clustering. Experiments on Swiss job advertisement datasets using GPT-4o, Gemini-2.0-flash, and DeepSeek-V3 show that the pipeline effectively reduces noise and manual curation effort: Turbo filtering lowered false positives by 82%, and aggregation reduced the number of items requiring review by 97%. Among the tested models, Gemini-2.0-flash achieved the highest precision, reaching a novelty detection ratio of up to 73% in the qualitative evaluation. These findings demonstrate the pipeline’s potential as an efficient tool for maintaining dynamic skill ontologies.
Anthology ID:
2026.lrec-1.611
Volume:
Proceedings of the Fifteenth Language Resources and Evaluation Conference
Month:
May
Year:
2026
Address:
Palma de Mallorca, Spain
Editors:
Stelios Piperidis, Núria Bel, Henk van den Heuvel, Nancy Ide, Simon Krek, Antonio Toral
Venue:
LREC
SIG:
Publisher:
ELRA Language Resource Association
Note:
Pages:
7701–7706
Language:
External URL:
https://lrec.elra.info/lrec2026-main-611
DOI:
10.63317/3twvjs5vmuwt
Bibkey:
Cite (ACL):
Gian Seifert and Simon Clematide. 2026. A Scalable Pipeline for Novelty Detection in Skill Extraction Using Large Language Models. In Proceedings of the Fifteenth Language Resources and Evaluation Conference, pages 7701–7706, Palma de Mallorca, Spain. ELRA Language Resource Association.
Cite (Informal):
A Scalable Pipeline for Novelty Detection in Skill Extraction Using Large Language Models (Seifert & Clematide, LREC 2026)
Copy Citation: