Voice agents face a fundamental tension: the reasoning, retrieval, and tool use that make foundation models capable are iterative and slow, while conversational interaction demands prompt responses. Smaller, real-time models meet the latency bar but do not match foundation models on complex tasks, leaving current voice agents to trade off responsiveness and capability. We introduce conversational infill, where a small Talker model both fluently integrates streamed Reasoner model knowledge into its responses during inference and immediately generates contextually grounded responses to hide external Reasoner latency. We curate a 290,571-example dataset and demonstrate that this task is learnable across seven small language models (135M to 1.7B parameters).

Our system, ConvFill, runs on an Apple M2 with millisecond-level latency, while remaining within 6.3% of corresponding frontier Reasoner performance. Conversational infill unlocks a new point on the latency–capability Pareto frontier, offering a practical path toward responsive and capable voice agents.