Multimodal harmful meme detection is challenging because harmful intent is often expressed through implicit image-text associations, cultural references, rhetorical framing, and context-dependent targets. Existing methods mainly rely on supervised multimodal models, MLLM prompting, or fixed agent workflows. However, they still lack experience accumulation and rely on static skill design, making it difficult to reuse prior judgments or adapt to emerging meme templates and long-tail harm patterns. To address these limitations, we propose HarmEvoSkill (Harmful Meme Evolving Skill-Memory), a label-free self-evolving Skill-Memory framework for harmful meme detection. HarmEvoSkill converts unlabeled reference memes into Case Memory, Insight Memory, and Risk Memory. It then automatically discovers candidate skills, abstracts recurring reasoning patterns, and induces executable skill cards without predefined skill labels. During inference, a Skill Planner retrieves relevant memories, selects task-specific skills, and generates structured evidence packets. A graph-assisted judge organizes the evidence into a harmful–harmless claim graph, makes the final prediction, and updates memory through graph diagnosis. Extensive benchmark evaluations show that HarmEvoSkill achieves consistently strong performance among evaluated methods. On HarM, it improves Accuracy by 9.33% and Macro-F1 by 17.97% over the direct MLLM baseline.