Contemporary large language models (LLMs) exhibit multilingual capabilities but remain predominantly English-centric, resulting in a persistent performance gap across languages. While some approaches to multilingual training are grounded in the implicit pivot language hypothesis to improve interpretability, it remains underexplored how the languages represented in each layer are affected during training, particularly in practical, cost-efficient instruction-tuning scenarios. In this study, we investigate how instruction tuning with Low-Rank Adaptation (LoRA) in a target language affects the layer-wise language representation of instruction-tuned LLMs, with the aim of assessing the relevance of the implicit pivot language hypothesis as a theoretical grounding for multilingual finetuning. We find that target-language instruction tuning shifts models’ internal language representation in a non-uniform manner across layers and sublayers: middle layers shift further toward English, while early and final layers primarily serve the mapping between the target language and the dominant intermediate language. Our findings empirically support the implicit pivot language hypothesis in the context of instruction tuning, demonstrating that layer-wise parameter variation broadly corresponds to the hypothesized functional roles of layers in multilingual processing. Our code is available at: https://github.com/hyemijo/layer_interpretability

Instruction Tuning on Large Language Models is an essential process for model to function well and achieve high performance in the specific tasks. Accordingly, in mainstream languages such as English, instruction-based datasets are being constructed and made publicly available. In the case of Korean, publicly available models and datasets all rely on using the output of ChatGPT or translating datasets built in English. In this paper, We introduce KIT-19 as an instruction dataset for the development of LLM in Korean. KIT-19 is a dataset created in an instruction format, comprising 19 existing open-source datasets for Korean NLP tasks. In this paper, we train a Korean Pretrained LLM using KIT-19 to demonstrate its effectiveness. The experimental results show that the model trained on KIT-19 significantly outperforms existing Korean LLMs. Based on the its quality and empirical results, this paper proposes that KIT-19 has the potential to make a substantial contribution to the future improvement of Korean LLMs’ performance.