Fragment-based drug discovery optimizes small molecular hits through three recurring transformations: fragment growing, linking, and merging. Existing generative methods, however, usually specialize in a single transformation and have focused primarily on linker design. We introduce FragmentGPT, a unified chemical language model that casts all three operations as conditional molecular generation. To build this capability, we construct a chemically-aware pre-training corpus using energy-guided bond cleavage, together with overlap-based examples that teach the model to connect separated fragments and consolidate shared substructures. We further introduce Reward-Ranked Alignment with Expert Exploration (RAE), which combines expert-guided candidate generation, Pareto- and composite-reward-based selection, and supervised fine-tuning for multi-objective molecular optimization. Conditioned on one or more fragments, FragmentGPT can expand a scaffold, connect distinct subunits, or merge overlapping structures into a complete molecule. Experiments on RTCB and SARS-CoV-2 main protease (3CLPro), together with ablation studies, show high chemical validity and strong aggregate performance across multiple drug-design objectives.