Cross-Lingual Transfer of Acceptability Judgement
LING-UA 52-001 Machine Learning for Language Understanding (NYU Courant Institute of Mathematical Sciences)
Spring 2021 Group Project of 2
Many recent studies have attempted to evaluate the grammatical knowledge acquired by language models. While most of these studies focus on English, models trained on multi-lingual corpora have also demonstrated success in transferring grammatical knowledge between languages (Mueller et al., 2020). Many studies on the transfer of grammatical knowledge evaluate models such as LSTMs and mBERT (Devlin et al., 2019), yet few have studied the grammatical abilities of the new state-of-the-art XLM-R model (Conneau et al., 2020). In this paper, we evaluate the ability of XLM-R to learn and transfer grammatical knowledge from a source language (English) to 4
% similar and dissimilar
target languages (German, Hebrew, French and Russian) using the datasets created by Mueller et al., 2020. Furthermore, we test the model on a low-resource language (Nepali). The fine-tuned model achieves a perfect accuracy in English and an average accuracy greater that 80\% in every other language. In Nepali, its accuracy on simple agreement is greater that 85\%. Our results demonstrate that XLM-R can successfully transfer grammatical knowledge from English to other languages. We also find evidence that the average accuracy of each language is correlated with the size of the corpora used for that language during the pre-training of XLM-R.