Automatic Language Identification in Texts

A Survey

Tommi Jauhiainen, Marco Lui, Marcos Zampieri, Timothy Baldwin, Krister Lindén

Tutkimustuotos: ArtikkelijulkaisuArtikkeliTieteellinenvertaisarvioitu

Kuvaus

Language identification (LI) is the problem of determining the natural language that a document or part thereof is written in. Automatic LI has been extensively researched for over fifty years. Today, LI is a key part of many text processing pipelines, as text processing techniques generally assume that the language of the input text is known. Research in this area has recently been especially active. This article provides a brief history of LI research, and an extensive survey of the features and methods used so far in the LI literature. For describing the features and methods we introduce a unified notation. We discuss evaluation methods, applications of LI, as well as off-the-shelf LI systems that do not require training by the end user. Finally, we identify open issues, survey the work to date on each issue, and propose future directions for research in LI.
Alkuperäiskielienglanti
LehtiJournal of Artificial Intelligence Research
Vuosikerta65
Sivut675-782
Sivumäärä108
ISSN1076-9757
TilaJulkaistu - 25 elokuuta 2019
OKM-julkaisutyyppiA1 Alkuperäisartikkeli tieteellisessä aikakauslehdessä, vertaisarvioitu

Tieteenalat

  • 6121 Kielitieteet
  • 213 Sähkö-, automaatio- ja tietoliikennetekniikka, elektroniikka

Lainaa tätä

Jauhiainen, Tommi ; Lui, Marco ; Zampieri, Marcos ; Baldwin, Timothy ; Lindén, Krister. / Automatic Language Identification in Texts : A Survey. Julkaisussa: Journal of Artificial Intelligence Research. 2019 ; Vuosikerta 65. Sivut 675-782.
@article{dfa52552524846f4807bda9d45c086b1,
title = "Automatic Language Identification in Texts: A Survey",
abstract = "Language identification (LI) is the problem of determining the natural language that a document or part thereof is written in. Automatic LI has been extensively researched for over fifty years. Today, LI is a key part of many text processing pipelines, as text processing techniques generally assume that the language of the input text is known. Research in this area has recently been especially active. This article provides a brief history of LI research, and an extensive survey of the features and methods used so far in the LI literature. For describing the features and methods we introduce a unified notation. We discuss evaluation methods, applications of LI, as well as off-the-shelf LI systems that do not require training by the end user. Finally, we identify open issues, survey the work to date on each issue, and propose future directions for research in LI.",
keywords = "6121 Languages, 213 Electronic, automation and communications engineering, electronics",
author = "Tommi Jauhiainen and Marco Lui and Marcos Zampieri and Timothy Baldwin and Krister Lind{\'e}n",
year = "2019",
month = "8",
day = "25",
language = "English",
volume = "65",
pages = "675--782",
journal = "Journal of Artificial Intelligence Research",
issn = "1076-9757",
publisher = "Association for the Advancement of Artificial Intelligence (AAAI)",

}

Automatic Language Identification in Texts : A Survey. / Jauhiainen, Tommi; Lui, Marco; Zampieri, Marcos; Baldwin, Timothy; Lindén, Krister.

julkaisussa: Journal of Artificial Intelligence Research, Vuosikerta 65, 25.08.2019, s. 675-782.

Tutkimustuotos: ArtikkelijulkaisuArtikkeliTieteellinenvertaisarvioitu

TY - JOUR

T1 - Automatic Language Identification in Texts

T2 - A Survey

AU - Jauhiainen, Tommi

AU - Lui, Marco

AU - Zampieri, Marcos

AU - Baldwin, Timothy

AU - Lindén, Krister

PY - 2019/8/25

Y1 - 2019/8/25

N2 - Language identification (LI) is the problem of determining the natural language that a document or part thereof is written in. Automatic LI has been extensively researched for over fifty years. Today, LI is a key part of many text processing pipelines, as text processing techniques generally assume that the language of the input text is known. Research in this area has recently been especially active. This article provides a brief history of LI research, and an extensive survey of the features and methods used so far in the LI literature. For describing the features and methods we introduce a unified notation. We discuss evaluation methods, applications of LI, as well as off-the-shelf LI systems that do not require training by the end user. Finally, we identify open issues, survey the work to date on each issue, and propose future directions for research in LI.

AB - Language identification (LI) is the problem of determining the natural language that a document or part thereof is written in. Automatic LI has been extensively researched for over fifty years. Today, LI is a key part of many text processing pipelines, as text processing techniques generally assume that the language of the input text is known. Research in this area has recently been especially active. This article provides a brief history of LI research, and an extensive survey of the features and methods used so far in the LI literature. For describing the features and methods we introduce a unified notation. We discuss evaluation methods, applications of LI, as well as off-the-shelf LI systems that do not require training by the end user. Finally, we identify open issues, survey the work to date on each issue, and propose future directions for research in LI.

KW - 6121 Languages

KW - 213 Electronic, automation and communications engineering, electronics

M3 - Article

VL - 65

SP - 675

EP - 782

JO - Journal of Artificial Intelligence Research

JF - Journal of Artificial Intelligence Research

SN - 1076-9757

ER -