Task-oriented dialogue (TOD) systems are introduced to solve specific tasks, which focus on training multiple tasks such as language understanding, tracking states, and generating appropriate responses to help users achieve their specific goals. Currently, one of the remaining challenges in this emergent research field is the capability to produce more robust architectures fine-tuned for end-to-end TOD systems. In this study, we consider this issue by exploiting the ability of pre-trained models to provide synthesis responses, which are then used as the input for the fine-tuned process. The main idea is to overcome the gap between the training process and inference process during fine-tuning end-to-end TOD systems. The experiment on Multiwoz datasets shows the effectiveness of our model compared with strong baselines in this research field. The source code is available for further exploitation.