A common approach for teaching large language models (LLMs) to reason is to train on chains-of-thought (CoTs) of in-distribution reasoning problems. However, such annotated data is costly to obtain for every problem of interest. We want reasoning models to generalize beyond their training distribution, and ideally to generalize compositionally: they should combine atomic reasoning skills to solve harder unseen tasks. In this paper, we introduce a method to enable generalization to a target compositional task that has no labeled CoT data. We minimally modify CoT formats of constituent atomic tasks to be composable: we add prefixes to CoTs during training time, thereby encouraging the trained model to generate sequences of CoTs during inference. We train individual models on the atomic tasks with composable CoT data and combine them with multitask learning or model merging to address the target compositional task zero-shot. This model can be further trained on a small amount of compositional data using rejection sampling fine-tuning (RFT). Training LLMs on Composable CoT outperforms multitask learning and continued fine-tuning baselines on three domains of controlled compositional tasks and also yields improvement on math skill composition setting derived from the MATH dataset.