%0 Conference Proceedings %T Morph Call: Probing Morphosyntactic Content of Multilingual Transformers %A Mikhailov, Vladislav %A Serikov, Oleg %A Artemova, Ekaterina %Y Vylomova, Ekaterina %Y Salesky, Elizabeth %Y Mielke, Sabrina %Y Lapesa, Gabriella %Y Kumar, Ritesh %Y Hammarström, Harald %Y Vulić, Ivan %Y Korhonen, Anna %Y Reichart, Roi %Y Ponti, Edoardo Maria %Y Cotterell, Ryan %S Proceedings of the Third Workshop on Computational Typology and Multilingual NLP %D 2021 %8 June %I Association for Computational Linguistics %C Online %F mikhailov-etal-2021-morph %X The outstanding performance of transformer-based language models on a great variety of NLP and NLU tasks has stimulated interest in exploration of their inner workings. Recent research has been primarily focused on higher-level and complex linguistic phenomena such as syntax, semantics, world knowledge and common-sense. The majority of the studies is anglocentric, and little remains known regarding other languages, specifically their morphosyntactic properties. To this end, our work presents Morph Call, a suite of 46 probing tasks for four Indo-European languages of different morphology: Russian, French, English and German. We propose a new type of probing tasks based on detection of guided sentence perturbations. We use a combination of neuron-, layer- and representation-level introspection techniques to analyze the morphosyntactic content of four multilingual transformers, including their understudied distilled versions. Besides, we examine how fine-tuning on POS-tagging task affects the probing performance. %R 10.18653/v1/2021.sigtyp-1.10 %U https://aclanthology.org/2021.sigtyp-1.10 %U https://doi.org/10.18653/v1/2021.sigtyp-1.10 %P 97-121