Harmful meme detection requires a stable understanding of how the image and the text work together. Since the text is already given as external input, a natural test is whether predictions remain stable when the corresponding visible text is removed from the image. Motivated by test-time multi-view inference, we build a text-erased reference view by keeping the external text unchanged and removing the visible text. If the model uses the external text in a stable way, this change should not greatly affect its harmfulness prediction. However, we observe clear score shifts and prediction flips, showing cross-view instability under fixed external text. We further find that the cross-view change after neutral-probe filtering is related to error risk, so it can serve as a test-time diagnostic signal. Based on this finding, we propose TIDC, a test-time framework that uses the filtered signal to calibrate the original-view score. We also provide theoretical analysis to support the diagnostic signal and calibration design. Experiments on HarM, FHM, and MAMI show that TIDC improves zero-shot performance across frozen multimodal backbones, without parameter updates or reasoning chains.