Voice agents face a fundamental tension: the reasoning, retrieval, and tool use that make foundation models capable are iterative and slow, while conversational interaction demands prompt responses. Smaller, real-time models meet the latency bar but do not match foundation models on complex tasks, leaving current voice agents to trade off responsiveness and capability. We introduce conversational infill, where a small Talker model both fluently integrates streamed Reasoner model knowledge into its responses during inference and immediately generates contextually grounded responses to hide external Reasoner latency. We curate a 290,571-example dataset and demonstrate that this task is learnable across seven small language models (135M to 1.7B parameters).

Our system, ConvFill, runs on an Apple M2 with millisecond-level latency, while remaining within 6.3% of corresponding frontier Reasoner performance. Conversational infill unlocks a new point on the latency–capability Pareto frontier, offering a practical path toward responsive and capable voice agents.

While NLP research has made strides in conversational tasks, many approaches focus on single-turn responses with well-defined objectives or evaluation criteria. In contrast, coaching presents unique challenges with initially undefined goals that evolve through multi-turn interactions, subjective evaluation criteria, mixed-initiative dialogue. In this work, we describe and implement five multi-turn coaching agents that exhibit distinct conversational styles, and evaluate them through a user study, collecting first-person feedback on 155 conversations. We find that users highly value core functionality, and that stylistic components in absence of core components are viewed negatively. By comparing user feedback with third-person evaluations from health experts and an LM, we reveal significant misalignment across evaluation approaches. Our findings provide insights into design and evaluation of conversational coaching agents and contribute toward improving human-centered NLP applications.