Language Model Adaptation for Language and Dialect Identification of Text

Research output: Contribution to journalArticleScientificpeer-review

Abstract

This article describes an unsupervised language model (LM) adaptation approach that can be used to enhance the performance of language identification methods. The approach is applied to a current version of the HeLI language identification method, which is now called HeLI 2.0. We describe the HeLI 2.0 method in detail. The resulting system is evaluated using the datasets from the German dialect identifi- cation and Indo-Aryan language identification shared tasks of the VarDial workshops 2017 and 2018. The new approach with LM adaptation provides considerably higher F1-scores than the basic HeLI or HeLI 2.0 methods or the other systems which participated in the shared tasks. The results indicate that unsu- pervised LM adaptation should be considered as an option in all language identification tasks, especially in those where encountering out-of-domain data is likely.
Original languageEnglish
Article number135132491900038
JournalNatural Language Engineering
Volume25
Issue number5
Pages (from-to)561-583
Number of pages23
ISSN1351-3249
DOIs
Publication statusPublished - Sep 2019
MoE publication typeA1 Journal article-refereed

Fields of Science

  • 6121 Languages

Cite this