Memes are widely used not only as static artifacts but also as interactive replies in online conversations. While computational research has focused on analyzing intrinsic properties of memes, the dynamic and contextual use of memes to create humor remains largely unaddressed. We introduce the Meme Reply Selection task and present MaMe-Re, a benchmark of 100,000 human-annotated context–meme pairs built from openly licensed Japanese manga panels. Our analysis reveals three key findings: (1) large language models show modest advantages over embedding methods by capturing social cues such as irony and exaggeration beyond surface-level semantic matching; (2) visual information does not improve performance, revealing a gap between understanding visual content and effectively using it for contextual humor; (3) while models can match human judgments when the humorous option is clearly distinguishable, they struggle to distinguish subtle differences in wit among semantically similar candidates. These findings establish contextually humorous reply selection as an open challenge for current models.