Language Model Adaptation for Language and Dialect Identification of Text

Tommi Sakari Jauhiainen; Bo Krister Johan Linden; Heidi Annika Jauhiainen

doi:10.1017/S135132491900038X

Language Model Adaptation for Language and Dialect Identification of Text

Research output: Contribution to journal › Article › Scientific › peer-review

Abstract

This article describes an unsupervised language model (LM) adaptation approach that can be used to enhance the performance of language identification methods. The approach is applied to a current version of the HeLI language identification method, which is now called HeLI 2.0. We describe the HeLI 2.0 method in detail. The resulting system is evaluated using the datasets from the German dialect identifi- cation and Indo-Aryan language identification shared tasks of the VarDial workshops 2017 and 2018. The new approach with LM adaptation provides considerably higher F1-scores than the basic HeLI or HeLI 2.0 methods or the other systems which participated in the shared tasks. The results indicate that unsu- pervised LM adaptation should be considered as an option in all language identification tasks, especially in those where encountering out-of-domain data is likely.

Original language	English
Article number	135132491900038
Journal	Natural Language Engineering
Volume	25
Issue number	5
Pages (from-to)	561-583
Number of pages	23
ISSN	1351-3249
DOIs	https://doi.org/10.1017/S135132491900038X
Publication status	Published - Sept 2019
MoE publication type	A1 Journal article-refereed

Fields of Science

6121 Languages

Access to Document

10.1017/S135132491900038XLicence: CC BY

LanguageModelAdaptationFinal published version, 333 KBLicence: CC BY

ANEE: Centre of Excellence in Ancient Near Eastern Empires
Silverman, J. M. (Principal Investigator), Svärd, S. (Project manager), Lahelma, A. (Principal Investigator), Nikki, N. (Participant), Töyräänvuori, J. (Participant), Jokiranta, J. (Participant), Laine, L. (Participant), Kletter, R. (Participant), Bonnie, R. (Participant), Thomas, S. (Participant), Holmqvist-Sipilä, E. (Participant), Kouki, P. (Participant), Wasmuth, M. (Participant), Wallis, C. (Participant), Lorenzon, M. (Participant), Jauhiainen, H. (Participant), Jauhiainen, T. (Participant), Alstola, T. (Participant), Bach, J. (Participant), Fink, S. (Participant), Konstantopoulos, G. (Participant), Zaia, S. (Participant), Sahala, A. (Participant), Miettunen, P. H. (Participant), Avneri Meir, R. (Participant), Aissaoui, A. I. (Participant), Arpe, O. P. (Participant), Luukko, M. I. (Participant), Kovanen, H. (Participant), Linden, K. (Participant), Bennett, E. R. (Participant), Saari, S. (Participant), Vanderstraeten, E. (Participant), Autere, K. (Participant), Debourse, C. H. I. (Participant), Kailamäki, U. (Participant), Kautonen, S. K. (Participant), Spunaugle, A. (Participant), Tambs, L. (Participant), Jones, C. W. (Participant), Wijnsma, U. Z. (Participant), Valk, J. (Participant), Gheorghiade, P. (Participant), Autere, K. (Participant), Uotila, R. (Participant), Reinikainen, S. J. J. (Participant), Cutillas Victoria, B. (Participant), Holappa, M. (Participant), Kletter, R. (Participant), Smith, S. L. (Participant) & Pfoh, E. (Participant)
01/01/2018 → 31/12/2025
Project: Research project
FIN-CLARIN UPGRADE - CLARINin vahvistaminen Suomessa FIN-CLARIN UPGRADE - Förstärkning av CLARIN i Finland
Linden, K. (Principal Investigator), Lennes, M. (Project manager), Westerlund, H. (Project manager), Piitulainen, J. (Participant), Niemi, J. (Participant), Axelson, E. (Participant), Jauhiainen, T. (Participant), Drobac, S. (Participant), Hardwick, S. (Participant), Kauppinen, P. (Participant), Voutilainen, A. (Project manager), Dieckmann, U. (Participant), Palolahti, M. (Participant), Matthiesen, M. (Project manager) & Aalto, T. (Participant)
01/01/2017 → 31/12/2019
Project: Other project

Cite this

@article{6798b31a579b4457aa592463841f39fd,

title = "Language Model Adaptation for Language and Dialect Identification of Text",

abstract = "This article describes an unsupervised language model (LM) adaptation approach that can be used to enhance the performance of language identification methods. The approach is applied to a current version of the HeLI language identification method, which is now called HeLI 2.0. We describe the HeLI 2.0 method in detail. The resulting system is evaluated using the datasets from the German dialect identifi- cation and Indo-Aryan language identification shared tasks of the VarDial workshops 2017 and 2018. The new approach with LM adaptation provides considerably higher F1-scores than the basic HeLI or HeLI 2.0 methods or the other systems which participated in the shared tasks. The results indicate that unsu- pervised LM adaptation should be considered as an option in all language identification tasks, especially in those where encountering out-of-domain data is likely.",

keywords = "6121 Languages",

author = "Jauhiainen, \{Tommi Sakari\} and Linden, \{Bo Krister Johan\} and Jauhiainen, \{Heidi Annika\}",

year = "2019",

month = sep,

doi = "10.1017/S135132491900038X",

language = "English",

volume = "25",

pages = "561--583",

journal = "Natural Language Engineering",

issn = "1351-3249",

publisher = "Cambridge University Press",

number = "5",

}

TY - JOUR

T1 - Language Model Adaptation for Language and Dialect Identification of Text

AU - Jauhiainen, Tommi Sakari

AU - Linden, Bo Krister Johan

AU - Jauhiainen, Heidi Annika

PY - 2019/9

Y1 - 2019/9

N2 - This article describes an unsupervised language model (LM) adaptation approach that can be used to enhance the performance of language identification methods. The approach is applied to a current version of the HeLI language identification method, which is now called HeLI 2.0. We describe the HeLI 2.0 method in detail. The resulting system is evaluated using the datasets from the German dialect identifi- cation and Indo-Aryan language identification shared tasks of the VarDial workshops 2017 and 2018. The new approach with LM adaptation provides considerably higher F1-scores than the basic HeLI or HeLI 2.0 methods or the other systems which participated in the shared tasks. The results indicate that unsu- pervised LM adaptation should be considered as an option in all language identification tasks, especially in those where encountering out-of-domain data is likely.

AB - This article describes an unsupervised language model (LM) adaptation approach that can be used to enhance the performance of language identification methods. The approach is applied to a current version of the HeLI language identification method, which is now called HeLI 2.0. We describe the HeLI 2.0 method in detail. The resulting system is evaluated using the datasets from the German dialect identifi- cation and Indo-Aryan language identification shared tasks of the VarDial workshops 2017 and 2018. The new approach with LM adaptation provides considerably higher F1-scores than the basic HeLI or HeLI 2.0 methods or the other systems which participated in the shared tasks. The results indicate that unsu- pervised LM adaptation should be considered as an option in all language identification tasks, especially in those where encountering out-of-domain data is likely.

KW - 6121 Languages

U2 - 10.1017/S135132491900038X

DO - 10.1017/S135132491900038X

M3 - Article

SN - 1351-3249

VL - 25

SP - 561

EP - 583

JO - Natural Language Engineering

JF - Natural Language Engineering

IS - 5

M1 - 135132491900038

ER -

Language Model Adaptation for Language and Dialect Identification of Text

Abstract

Fields of Science

Access to Document

Projects

ANEE: Centre of Excellence in Ancient Near Eastern Empires

FIN-CLARIN UPGRADE - CLARINin vahvistaminen Suomessa FIN-CLARIN UPGRADE - Förstärkning av CLARIN i Finland

Cite this