Multi-agent LLM systems increasingly rely on persistent memory to coordinate role-specialized work. Correctness requires more than recalling a fact: the fact must reach the right role, remain hidden from unauthorized roles, carry the right source, and update when a more authoritative value appears. We introduce COLLABMEM, a diagnostic benchmark for collaborative memory operations. Its matched memory-only harness fixes the trace, target role, answerer, judge, and gold event, localizing errors to sharing, access, routing, provenance, conflict resolution, or retention. COLLABMEM contains 58 hand-authored scenarios (1,118 core probes), a 174-scenario scaled split (3,359 probes), adversarial and compositional hard sets, and six transactive-memory competencies. We also propose TMS-GRAPH++, a scope-aware graph memory framework with role-filtered views, writer/provenance structure, and temporal [CURRENT] marking. Across five evaluation splits, TMS-GRAPH++ achieves the highest non-Oracle average score (69.6%). Evaluations using the official SDKs of Mem0, Letta, and A-Mem show that shared archives help, but reliable collaborative memory still requires explicit visibility, provenance, authority, and retention policies. We will release the dataset, evaluation code, and reproduced outputs to support future work on multi-agent memory systems.

The paper presents KMatrix-2, an open-source toolkit that supports comprehensive heterogeneous knowledge collaborative enhancement for Large Language Models (LLMs). As the successor of KMatrix, our toolkit offers powerful modular components and typical enhancement patterns for convenient construction of mainstream knowledge-enhanced LLMs systems. Besides, it provides unified knowledge integration and joint knowledge retrieval methods to achieve more comprehensive heterogeneous knowledge collaborative enhancement. Compared with KMatrix which mainly focuses on descriptive knowledge, this work additionally considers procedural knowledge. Moreover, systematic inter-context and context-memory knowledge conflict resolution methods are offered for better knowledge integration. Some key research questions in heterogeneous knowledge-enhanced Large Language Models systems are analyzed, and our toolkit’s capability in building such systems is validated.

Knowledge-Enhanced Large Language Models (K-LLMs) system enhances Large Language Models (LLMs) abilities using external knowledge. Existing K-LLMs toolkits mainly focus on free-textual knowledge, lacking support for heterogeneous knowledge like tables and knowledge graphs, and fall short in comprehensive datasets, models, and user-friendly experience. To address this gap, we introduce KMatrix: a flexible heterogeneous knowledge enhancement toolkit for LLMs including verbalizing-retrieval and parsing-query methods. Our modularity and control-logic flow diagram design flexibly supports the entire lifecycle of various complex K-LLMs systems, including training, evaluation, and deployment. To assist K-LLMs system research, a series of related knowledge, datasets, and models are integrated into our toolkit, along with performance analyses of K-LLMs systems enhanced by different types of knowledge. Using our toolkit, developers can rapidly build, evaluate, and deploy their own K-LLMs systems.