In LLMs, the source of information used to produce an answer is necessary to explain the generation process.This work considers the case where the answer can be extracted from the question’s context ( contextual knowledge) or from the model weights (parametric knowledge). While methods have been developed to uncover the grounding sources of LLMs in question answering, these approaches mainly rely on the text modality, whereas most tasks involve multiple modalities. This study proposes to uncover LLMs’ internal signals to determine whether they use parametric or contextual knowledge in a bimodal setting. To this end, we create a counterfactual dataset to evaluate whether the model relies on parametric or contextual knowledge on text and/or image-based questions. We leverage probing methods based on the internal attention and the different latent representations to determine the model’s source of information. We evaluate the performances of the probing methods, focusing on the different modalities. We show that models share common signals across the corpus and modalities. This study opens the way to develop controllable methods by uncovering common signals in LLMs to determine the knowledge used in generation.