Language model (LM)-powered agents have shown promise in addressing complex embodied tasks by leveraging the commonsense world knowledge of pre-trained models in interactive physical environments. Yet, real-world deployment requires agents to continually adapt to a stream of tasks, where neither large-scale expert supervision nor reliable external feedback is consistently available. In this paper, we present ASC, a framework for efficient continual embodied task adaptation through amortizing self-correction. Specifically, ASC trains the policy under two objectives: a standard CoT reasoning objective distilled from stronger experts, and a reflexive reasoning objective that internalizes self-correction by correcting initial rationales generated by a previously learned policy. This dual optimization allows the agent to adapt more efficiently through forward knowledge transfer over time, ultimately fostering robust CoT reasoning at inference without requiring explicit reflection. Furthermore, we introduce rationale-wise test-time scaling that dynamically adjusts the depth of CoT reasoning based on the model’s confidence in actions inferred from its own rationales. Through experiments on VirtualHome and ALFWorld, we demonstrate performance superiority over other LM-based planning and continual task adaptation approaches, while achieving strong efficiency in computation, data usage.