Controllable text-to-speech (TTS) aims to generate speech from text with precise control over prosodic and speaker-related attributes such as pitch, age, and accent. Existing methods mainly rely on natural language prompts or reference audio cloning. However, prompt-based approaches often suffer from the cross-modal semantic gap between textual descriptions and target speech attributes, resulting in imprecise and coarse-grained control, while cloning-based methods are constrained by reference audio and offer limited flexibility. To address these limitations, we propose TTS-Hub, a controllable TTS framework that leverages modular Low-Rank Adaptation (LoRA) components and their arithmetic composition for fine-grained and flexible speech control. Specifically, we first build a comprehensive Data Hub covering 6 high-level categories and 32 fine-grained speech attributes, and use it to construct a LoRA Hub in which each LoRA is specialized for a specific attribute. During inference, TTS-Hub selects and composes the required LoRA modules through simple arithmetic operations, enabling flexible and extensible attribute control without retraining the backbone model. Experiments on two mainstream TTS frameworks demonstrate that TTS-Hub consistently outperforms strong baselines and enables precise and flexible control.