Diffusion models have shown impressive performance in many domains. However, the model’s capability to follow natural language instructions (e.g., spatial relationships between objects, generating complex scenes) is still unsatisfactory. In this work, we propose Iterative Prompt Relabeling (IPR), a novel algorithm that aligns images to text through iterative image sampling and prompt relabeling with feedback. IPR first samples a batch of images conditioned on the text, then relabels the text prompts of unmatched text-image pairs with classifier feedback. We conduct thorough experiments on SDv2 and SDXL, testing their capability to follow instructions on spatial relations. With IPR, we improved up to 15.22% (absolute improvement) on the challenging spatial relation VISOR benchmark, demonstrating superior performance compared to previous RL methods. Our code is publicly available at https://github.com/cxy000000/IPR-RLDF.

Сontent and language integrated learning is considered a powerful tool to promote inclusion in educational settings of learners for whom the language of instruction is their additional language. Language-related difficulties of those learners have been claimed detrimental for attaining personal educational goals. Academic language places increased cognitive demands on the learning process in general due to 1) its internal complexity; 2) L2 speakers’ lower proficiency; 3) their disadvantage in terms of real-time processing. Facilitators are, therefore, encouraged to integrate interactional CLIL-elements (e.g., scaffolding) during content instruction that provide the necessary pedagogical support for better understanding of disciplinary concepts and their interrelation. In the current contribution, we present the concept and first results of Rail.lexis, a collaborative project of the Department of German Studies and the Department of Railway Engineering at TU Brauschweig. We present and discuss several conversational arrangements (e.g., word guessing games, a differential task matrix) that were designed to engage the learners of heterogeneous linguistic backgrounds in meaningful interactions in subject-specific classes. Subject-specific tasks are gradient regarding their cognitive complexity and the background knowledge required to solve them. Therefore, the linguistic repertoire required to negotiate different task types is also differential to ensure the participation of linguistically diverse students in language-enhanced classroom interactions.