Automatic Language Identification in Texts: A Survey

Tommi Jauhiainen, Marco Lui, Marcos Zampieri, Timothy Baldwin, Krister Lindén

Forskningsoutput: TidskriftsbidragArtikelVetenskapligPeer review

Sammanfattning

Language identification (LI) is the problem of determining the natural language that a document or part thereof is written in. Automatic LI has been extensively researched for over fifty years. Today, LI is a key part of many text processing pipelines, as text processing techniques generally assume that the language of the input text is known. Research in this area has recently been especially active. This article provides a brief history of LI research, and an extensive survey of the features and methods used so far in the LI literature. For describing the features and methods we introduce a unified notation. We discuss evaluation methods, applications of LI, as well as off-the-shelf LI systems that do not require training by the end user. Finally, we identify open issues, survey the work to date on each issue, and propose future directions for research in LI.
Originalspråkengelska
TidskriftJournal of Artificial Intelligence Research
Volym65
Sidor (från-till)675-782
Antal sidor108
ISSN1076-9757
StatusPublicerad - 25 aug 2019
MoE-publikationstypA1 Tidskriftsartikel-refererad

Vetenskapsgrenar

  • 6121 Språkvetenskaper
  • 213 El-, automations- och telekommunikationsteknik, elektronik

Citera det här

Jauhiainen, Tommi ; Lui, Marco ; Zampieri, Marcos ; Baldwin, Timothy ; Lindén, Krister. / Automatic Language Identification in Texts : A Survey. I: Journal of Artificial Intelligence Research. 2019 ; Vol. 65. s. 675-782.
@article{dfa52552524846f4807bda9d45c086b1,
title = "Automatic Language Identification in Texts: A Survey",
abstract = "Language identification (LI) is the problem of determining the natural language that a document or part thereof is written in. Automatic LI has been extensively researched for over fifty years. Today, LI is a key part of many text processing pipelines, as text processing techniques generally assume that the language of the input text is known. Research in this area has recently been especially active. This article provides a brief history of LI research, and an extensive survey of the features and methods used so far in the LI literature. For describing the features and methods we introduce a unified notation. We discuss evaluation methods, applications of LI, as well as off-the-shelf LI systems that do not require training by the end user. Finally, we identify open issues, survey the work to date on each issue, and propose future directions for research in LI.",
keywords = "6121 Languages, 213 Electronic, automation and communications engineering, electronics",
author = "Tommi Jauhiainen and Marco Lui and Marcos Zampieri and Timothy Baldwin and Krister Lind{\'e}n",
year = "2019",
month = "8",
day = "25",
language = "English",
volume = "65",
pages = "675--782",
journal = "Journal of Artificial Intelligence Research",
issn = "1076-9757",
publisher = "Association for the Advancement of Artificial Intelligence (AAAI)",

}

Automatic Language Identification in Texts : A Survey. / Jauhiainen, Tommi; Lui, Marco; Zampieri, Marcos; Baldwin, Timothy; Lindén, Krister.

I: Journal of Artificial Intelligence Research, Vol. 65, 25.08.2019, s. 675-782.

Forskningsoutput: TidskriftsbidragArtikelVetenskapligPeer review

TY - JOUR

T1 - Automatic Language Identification in Texts

T2 - A Survey

AU - Jauhiainen, Tommi

AU - Lui, Marco

AU - Zampieri, Marcos

AU - Baldwin, Timothy

AU - Lindén, Krister

PY - 2019/8/25

Y1 - 2019/8/25

N2 - Language identification (LI) is the problem of determining the natural language that a document or part thereof is written in. Automatic LI has been extensively researched for over fifty years. Today, LI is a key part of many text processing pipelines, as text processing techniques generally assume that the language of the input text is known. Research in this area has recently been especially active. This article provides a brief history of LI research, and an extensive survey of the features and methods used so far in the LI literature. For describing the features and methods we introduce a unified notation. We discuss evaluation methods, applications of LI, as well as off-the-shelf LI systems that do not require training by the end user. Finally, we identify open issues, survey the work to date on each issue, and propose future directions for research in LI.

AB - Language identification (LI) is the problem of determining the natural language that a document or part thereof is written in. Automatic LI has been extensively researched for over fifty years. Today, LI is a key part of many text processing pipelines, as text processing techniques generally assume that the language of the input text is known. Research in this area has recently been especially active. This article provides a brief history of LI research, and an extensive survey of the features and methods used so far in the LI literature. For describing the features and methods we introduce a unified notation. We discuss evaluation methods, applications of LI, as well as off-the-shelf LI systems that do not require training by the end user. Finally, we identify open issues, survey the work to date on each issue, and propose future directions for research in LI.

KW - 6121 Languages

KW - 213 Electronic, automation and communications engineering, electronics

M3 - Article

VL - 65

SP - 675

EP - 782

JO - Journal of Artificial Intelligence Research

JF - Journal of Artificial Intelligence Research

SN - 1076-9757

ER -