Mohammad Reza Taesiri
Author directory2026
SketchVLM: Vision language models can annotate images to explain thoughts and guide users
Brandon Collins | Logan Bolton | Hung Nguyen | Mohammad Reza Taesiri | Trung Bui | Anh Totti Nguyen
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Brandon Collins | Logan Bolton | Hung Nguyen | Mohammad Reza Taesiri | Trung Bui | Anh Totti Nguyen
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
When answering questions about images, humans naturally point, label, and draw to explain their reasoning. In contrast, modern vision–language models (VLMs) such as Gemini-3-Pro and GPT-5 only respond with text, which can be difficult for users to verify. We present SketchVLM, a training-free, model-agnostic framework that enables VLMs to produce non-destructive, editable SVG overlays on the input image to visually explain their answers. Across seven benchmarks spanning visual reasoning (maze navigation, ball-drop trajectory prediction, and object counting) and drawing (part labeling, connecting-the-dots, and drawing shapes around objects), SketchVLM improves visual reasoning task accuracy by up to +28.5 percentage points and annotation quality by up to 1.48x relative to image-editing and fine-tuned sketching baselines, while also producing annotations that are more faithful to the model’s stated answer. We find that single-turn generation already achieves strong accuracy and annotation quality, and multi-turn generation opens up further opportunities for human-AI collaboration. An interactive demo can be found at https://sketchvlm.github.io/.
2024
Fine-tuning Language Models for Triple Extraction with Data Augmentation
Yujia Zhang | Tyler Sadler | Mohammad Reza Taesiri | Wenjie Xu | Marek Reformat
Proceedings of the 1st Workshop on Knowledge Graphs and Large Language Models (KaLLM 2024)
Yujia Zhang | Tyler Sadler | Mohammad Reza Taesiri | Wenjie Xu | Marek Reformat
Proceedings of the 1st Workshop on Knowledge Graphs and Large Language Models (KaLLM 2024)
Advanced language models with impressive capabilities to process textual information can more effectively extract high-quality triples, which are the building blocks of knowledge graphs. Our work examines language models’ abilities to extract entities and the relationships between them. We use a diverse data augmentation process to fine-tune large language models to extract triples from the text. Fine-tuning is performed using a mix of trainers from HuggingFace and five public datasets, such as different variations of the WebNLG, SKE, DocRed, FewRel, and KELM. Evaluation involves comparing model outputs with test-set triples based on several criteria, such as type, partial, exact, and strict accuracy.The obtained results outperform ChatGPT and even match or exceed the performance of GPT-4.