Text-Attributed Graphs (TAGs), where each node is associated with text descriptions, are ubiquitous in real-world scenarios. They typically exhibit distinctive structure and domain-specific knowledge, motivating the development of Graph Foundation Models (GFMs) that generalize across diverse graphs and tasks. Despite large efforts to integrate Large Language Models (LLMs) and Graph Neural Networks (GNNs) for TAGs, most approaches suffer from decoupled architectures with two-stage alignment, limiting their synergistic potential. Even worse, existing methods assign out-of-vocabulary (OOV) tokens to graph nodes, leading to graph-specific semantics, token explosion, and incompatibility with task-oriented prompt templates, which hinders cross-graph and cross-task transferability. To address these challenges, we propose PromptGFM, a versatile GFM for TAGs grounded in graph vocabulary learning. PromptGFM comprises two key components: (a) a graph understanding module, which explicitly prompts LLMs to replicate the fine-grained GNN workflow within the text space, facilitating seamless GNN-LLM integration and elegant graph-text alignment; and (b) a graph inference module, which establishes a language-based graph vocabulary to ensure expressiveness, transferability, and scalability, enabling readable instructions for joint LLM fine-tuning across graphs and tasks. Extensive experiments demonstrate the superiority of PromptGFM, highlighting its strong transferability across diverse graphs and tasks and its adaptability to auxiliary pre-training tasks. The code is available at https://github.com/agiresearch/PromptGFM.