Reinforcement Learning (RL) has emerged as a dominant paradigm for eliciting long-horizon reasoning in Large Language Models (LLMs). However, scaling Tool-Integrated Reasoning (TIR) via RL remains challenging due to interaction collapse: a pathological state where models fail to sustain multi-turn tool usage, instead degenerating into heavy internal reasoning with only trivial, post-hoc code verification. We systematically study three questions: (i) how cold-start Supervised Fine-Tuning (SFT) induces an agentic, tool-using behavioral prior, (ii) how teacher-trajectory structure and interaction density shape exploration and downstream RL outcomes, and (iii) how the RL interaction budget affects learning dynamics and generalization under varying inference-time budgets. We then introduce ASTER (Agentic Scaling with Tool-integrated Extended Reasoning), a framework that circumvents this collapse through a targeted cold-start strategy prioritizing structured, interaction-dense trajectories. We find that a small expert cold-start set of just 4K structurally rich, interaction-dense trajectories yields the strongest downstream performance, establishing a robust prior that enables superior exploration during extended RL training. Extensive evaluations demonstrate that ASTER-4B achieves strong results on competitive mathematical benchmarks, reaching 89.6% mean accuracy on AIME2025 and remaining competitive with leading frontier open-source models. See more details at https://github.com/CodeByLeo-bit/ASTER.