Which Modality should I use - Text, Motif, or Image? : Understanding Graphs with Large Language Models

Debarati Das, Ishaan Gupta, Jaideep Srivastava, Dongyeop Kang


Abstract
Our research integrates graph data with Large Language Models (LLMs), which, despite their advancements in various fields using large text corpora, face limitations in encoding entire graphs due to context size constraints. This paper introduces a new approach to encoding a graph with diverse modalities, such as text, image, and motif, coupled with prompts to approximate a graph’s global connectivity, thereby enhancing LLMs’ efficiency in processing complex graph structures. The study also presents GraphTMI, a novel benchmark for evaluating LLMs in graph structure analysis, focusing on homophily, motif presence, and graph difficulty. Key findings indicate that the image modality, especially with vision-language models like GPT-4V, is superior to text in balancing token limits and preserving essential information and comes close to prior graph neural net (GNN) encoders. Furthermore, the research assesses how various factors affect the performance of each encoding modality and outlines the existing challenges and potential future developments for LLMs in graph understanding and reasoning tasks. Our code and data are publicly available on our project page - https://minnesotanlp.github.io/GraphLLM/
Anthology ID:
2024.findings-naacl.34
Volume:
Findings of the Association for Computational Linguistics: NAACL 2024
Month:
June
Year:
2024
Address:
Mexico City, Mexico
Editors:
Kevin Duh, Helena Gomez, Steven Bethard
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
503–519
Language:
URL:
https://aclanthology.org/2024.findings-naacl.34
DOI:
Bibkey:
Cite (ACL):
Debarati Das, Ishaan Gupta, Jaideep Srivastava, and Dongyeop Kang. 2024. Which Modality should I use - Text, Motif, or Image? : Understanding Graphs with Large Language Models. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 503–519, Mexico City, Mexico. Association for Computational Linguistics.
Cite (Informal):
Which Modality should I use - Text, Motif, or Image? : Understanding Graphs with Large Language Models (Das et al., Findings 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.findings-naacl.34.pdf
Copyright:
 2024.findings-naacl.34.copyright.pdf