We present Holmes-Graph, a framework to evaluate the linguistic competences of graph-based text embeddings generated by training graph neural networks (GNNs) on graph representations of text. The framework extends the Holmes benchmark, which previously covered only sparse, static, and transformer-based representations. The framework is flexible to be used with any graph representation and GNN architecture. Once weights are learned on a downstream task, frozen embeddings are probed across five linguistic competences. When applied to AI-generated text detection, Integrated Syntactic Graphs (ISG) combined with Graph Neural Networks (GNN) achieve an aggregate Holmes score of 0.662. This matches or exceeds DeBERTa-base on syntax, morphology, semantics, and reasoning, but not for discourse. The results confirm that graph-based text embeddings encode non-trivial linguistic knowledge and motivate their inclusion in future probing benchmarks.

The widespread availability of Large Language Models (LLMs) such as GPT-4 and Llama-3, among others, has led to a surge in machine-generated content across various platforms, including social media, educational tools, and academic settings. While these models demonstrate remarkable capabilities in generating coherent text, their misuse raises significant concerns. For this reason, detecting machine-generated text has become a pressing need to mitigate these risks. This research proposed a novel classification method combining text-graph representations with Graph Neural Networks (GNNs) and different node feature initialization strategies to distinguish between human-written and machine-generated content. Experimental results demonstrate that the proposed approach outperforms traditional machine learning classifiers, highlighting the effectiveness of integrating structural and semantic relationships in text.