On today’s social media, many argumentative artifacts appear in mixed image-text format. Meanwhile, the argument mining community has seen much success in developing argument understanding technologies in text. In this work, we bridge that gap by studying vision-language argument (VLA) parsing: generating an argument graph that captures both the explicit and implicit components of a vision-language argument. To lay the groundwork for this task, we annotate MemeArgs, a corpus of argument graphs for 614 persuasive memes. We then propose an automatic evaluation protocol for VLA parsing that accommodates its generative nature and follows standard argument mining practice. Evaluating current general-purpose vision-language models, we found that they all failed to perform knowledge-grounded visual perception reliably, which distorts their understanding of the arguments. VLA parsing thus presents an exciting challenge for multimodal argument understanding.