We propose RCEM, a Robust Conversational search EMbedder that augments text embedders with LLM query reformulation capabilities while approximately preserving their original generalization performance. Unlike prior conversational dense retrieval approaches that learn direct conversation-to-passage matching, RCEM aligns conversations, prepended by special token, to LLM-rewritten queries, while attempting to preserve the original embedding space. The approximately preserved embedding space automatically maps the rewritten queries to the relevant passages. As a result, RCEM (1) reduces overfitting by aligning conversations to shorter rewritten queries rather than full passages, (2) eliminates the need for conversation-to-passage relevance labels for training, and (3) approximately preserves the original embedding space, enabling conversational retrieval over indexes built by the base embedder without re-indexing. Extensive experiments show that RCEM consistently outperforms prior approaches, improving up to 30% retrieval accuracy under distributional shift.

We demonstrate an intelligent conversational agent system designed for advancing human-machine collaborative tasks. The agent is able to interpret a user’s communicative intent from both their verbal utterances and non-verbal behaviors, such as gestures. The agent is also itself able to communicate both with natural language and gestures, through its embodiment as an avatar thus facilitating natural symmetric multi-modal interactions. We demonstrate two intelligent agents with specialized skills in the Blocks World as use-cases of our system.