Fragkiskos Malliaros
Author directory2026
MUSART: A Scalable Multi-Domain Benchmark for Measuring Factual Recall in Large Language Models
Rajaa El Hamdani | Thomas Bonald | Fragkiskos Malliaros
Proceedings of the 11th Workshop on Automated Knowledge Base Construction (AKBC 2026)
Rajaa El Hamdani | Thomas Bonald | Fragkiskos Malliaros
Proceedings of the 11th Workshop on Automated Knowledge Base Construction (AKBC 2026)
Reliable evaluation of factual recall in large language models (LLMs) requires ground truth that is broad, clean, and complete. Yet existing benchmarks often rely on noisy extraction, narrow domains, or incomplete knowledge bases that penalize models for generating correct facts absent from the reference. We introduce MUSART, a benchmark of 112 596 factual triples over 31 domains, automatically constructed by pairing Wikipedia Good Article categories with Wikidata and selecting domain-salient relations via Personalized PageRank. To overcome knowledge-base incompleteness, we contribute an automated, character-grounded augmentation pipeline that mines verbatim facts from source articles. Evaluating twenty-one open-weight LLMs against both reference regimes reveals that (i) completeness correction systematically raises precision and lowers recall across all models, especially on multi-valued relations; and (ii) factual recall varies widely across subject domains, demonstrating why single-domain evaluation confounds model capability with subject matter. All code, data, and evaluation scripts are publicly released at https://github.com/Rajjaa/MUSART.
2018
Fusing Document, Collection and Label Graph-based Representations with Word Embeddings for Text Classification
Konstantinos Skianis | Fragkiskos Malliaros | Michalis Vazirgiannis
Proceedings of the Twelfth Workshop on Graph-Based Methods for Natural Language Processing (TextGraphs-12)
Konstantinos Skianis | Fragkiskos Malliaros | Michalis Vazirgiannis
Proceedings of the Twelfth Workshop on Graph-Based Methods for Natural Language Processing (TextGraphs-12)
Contrary to the traditional Bag-of-Words approach, we consider the Graph-of-Words(GoW) model in which each document is represented by a graph that encodes relationships between the different terms. Based on this formulation, the importance of a term is determined by weighting the corresponding node in the document, collection and label graphs, using node centrality criteria. We also introduce novel graph-based weighting schemes by enriching graphs with word-embedding similarities, in order to reward or penalize semantic relationships. Our methods produce more discriminative feature weights for text categorization, outperforming existing frequency-based criteria.