To address the limitations of conventional ICL, recent task-vector-based methods, such as Implicit ICL (I²CL), encode demonstrations into a unified task vector that is injected during inference, yielding promising improvements in efficiency and stability. However, existing methods encode each demonstration in isolation, neglecting the rich interactions between demonstrations and between demonstrations and queries—a core capability of standard ICL. This results in task vectors with limited expressiveness. To bridge this gap, we propose Interactive Implicit In-Context Learning (I³CL), a simple yet effective framework that restores these essential interactive capabilities. I³CL introduces only two lightweight attention-based modules, preserving the efficiency advantages of I²CL while delivering significant performance improvements. Experiments across nine diverse tasks using four LLMs show that I³CL achieves an average performance improvement of 3.54% over prior task-vector-based baselines. Comprehensive analyses on efficiency, scalability and task vector properties further confirm the superiority of our approach.