We introduce KASHAF, to our knowledge the first interpretable Semantic Textual Similarity (iSTS) resource and end-to-end system for Arabic: a corpus of 7,811 human-annotated sentence pairs (inter-annotator κ=0.83κ= 0.83 κ=0.83 on a doubly-annotated subset) under a three-level hierarchical schema (sentence, phrase, and word), together with a modular pipeline that produces this layered analysis plus a composite similarity score, the KASHAF Hierarchical Similarity Score (KHSS), calibrated to the ordinal SEM scale. On a held-out cross-batch test split (1,002 pairs, disjoint from training and dev), KASHAF reaches macro-F1 =0.659= 0.659 =0.659 (±0.016±0.016 ±0.016, 5 seeds) and Spearman ρ=0.6715ρ= 0.6715 ρ=0.6715; a rule-based antonym/negation polarity gate provides an empirical score-capping rule for high-similarity contradictions, and the MARBERTv2 backbone leads three alternative Arabic encoders. As an off-the-shelf diagnostic rather than a fine-tuned comparison, four contemporary LLMs under matched prompting reach at most macro-F1 0.215 (standard prompting) and 0.336 (chain-of-thought), a 32–44-point gap that reflects systematic over-prediction of SEM-PARA at the expense of the dominant SEM-PARTIAL class. We release the schema, guidelines, code, models, and a reproducibility pipeline; the corpus is available from KSAA on request.

Academic paraphrasing in Modern Standard Arabic must change expression without distorting scientific content. We study this balance through AraScholar, a corpus of 20,002 manually written source–paraphrase pairs with 522 raw domain labels and item-level provenance. On 2,000 held-out sentences, fine-tuned AraT5v2 favors lexical reuse under beam search, while frontier models produce more extensive rewrites. With corrected Arabic punctuation normalization, diverse decoding and preservation-constrained reranking attain 74.25% coverage of a fixed novelty–preservation band, compared with 52.60% for zero-shot GPT-4o. Gold coverage is only 46.65%, showing that this band measures constraint compliance rather than human quality. Post-hoc target matching substantially improves a calibrated GPT-4o control and narrows the gap, although candidate budgets differ. Crucially, an independent 200-instance audit favors frontier models on meaning and paraphrase quality. We document the corpus, decoding controls, negative distillation result, and metric sensitivity, including test-derived calibration and checkpoint-selection limitations. The findings establish operating-point control as a useful capability while demonstrating why calibration must be evaluated alongside independent assessments of writing quality.

The impressive advancement of Large Language Models (LLMs) in English has not been matched across all languages. In particular, LLM performance in Arabic lags behind, due to data scarcity, linguistic diversity of Arabic and its dialects, morphological complexity, etc. Progress is further hindered by the quality of Arabic benchmarks, which typically rely on static, publicly available data, lack comprehensive task coverage, or do not provide dedicated platforms with blind test sets. This makes it challenging to measure actual progress and to mitigate data contamination. Here, we aim to bridge these gaps. In particular, we introduce BALSAM, a comprehensive, community-driven benchmark aimed at advancing Arabic LLM development and evaluation. It includes 78 NLP tasks from 14 broad categories, with 52K examples divided into 37K test and 15K development, and a centralized, transparent platform for blind evaluation. We envision BALSAM as a unifying platform that sets standards and promotes collaborative research to advance Arabic LLM capabilities.

This paper investigates the effectiveness of retrieval-augmented generation (RAG) pipelines, focusing on the Arabic lexical information retrieval. Specifically, it analyzes how embedding models affect the recall of Arabic lexical information and evaluates the ability of large language models (LLMs) to produce accurate and contextually relevant answers within the RAG pipelines. We examine a dataset of over 88,000 words from the Riyadh dictionary and evaluate the models using metrics such as Top-K Recall, Mean Reciprocal Rank (MRR), F1 Score, Cosine Similarity, and Accuracy. The research assesses the capabilities of several embedding models, including E5-large, BGE, AraBERT, CAMeLBERT, and AraELECTRA, highlighting a disparity in performance between sentence embeddings and word embeddings. Sentence embedding with E5 achieved the best results, with a Top-5 Recall of 0.88, and an MRR of 0.48. For the generation models, we evaluated GPT-4, GPT-3.5, SILMA-9B, Gemini-1.5, Aya-8B, and AceGPT-13B based on their ability to generate accurate and contextually appropriate responses. GPT-4 demonstrated the best performance, achieving an F1 score of 0.90, an accuracy of 0.82, and a cosine similarity of 0.87. Our results emphasize the strengths and limitations of both embedding and generation models in Arabic tasks.

This paper outlines the KSAA-CAD shared task, highlighting the Contemporary Arabic Language Dictionary within the scenario of developing a Reverse Dictionary (RD) system and enhancing Word Sense Disambiguation (WSD) capabilities. The first KSAA-RD (Al-Matham et al., 2023) highlighted significant gaps in the domain of RDs, which are designed to retrieve words by their meanings or definitions. This shared task comprises two tasks: RD and WSD. The RD task focuses on identifying word embeddings that most accurately match a given definition, termed a “gloss,” in Arabic. Conversely, the WSD task involves determining the specific meaning of a word in context, particularly when the word has multiple meanings. The winning team achieved the highest-ranking score of 0.0644 in RD using Electra embeddings. In this paper, we describe the methods employed by the participating teams and provide insights into the future direction of KSAA-CAD.