Automatic Language Identification in Texts: A Survey

Tommi Jauhiainen, Marco Lui, Marcos Zampieri, Timothy Baldwin, Krister Lindén

Research output: Contribution to journalArticleScientificpeer-review

Abstract

Language identification (LI) is the problem of determining the natural language that a document or part thereof is written in. Automatic LI has been extensively researched for over fifty years. Today, LI is a key part of many text processing pipelines, as text processing techniques generally assume that the language of the input text is known. Research in this area has recently been especially active. This article provides a brief history of LI research, and an extensive survey of the features and methods used so far in the LI literature. For describing the features and methods we introduce a unified notation. We discuss evaluation methods, applications of LI, as well as off-the-shelf LI systems that do not require training by the end user. Finally, we identify open issues, survey the work to date on each issue, and propose future directions for research in LI.
Original languageEnglish
JournalJournal of Artificial Intelligence Research
Number of pages97
ISSN1076-9757
Publication statusAccepted/In press - 13 May 2019
MoE publication typeA1 Journal article-refereed

Bibliographical note

Under review at JAIR - Journal of Artificial Intelligence Research

Fields of Science

  • 6121 Languages
  • 213 Electronic, automation and communications engineering, electronics

Cite this

@article{dfa52552524846f4807bda9d45c086b1,
title = "Automatic Language Identification in Texts: A Survey",
abstract = "Language identification (LI) is the problem of determining the natural language that a document or part thereof is written in. Automatic LI has been extensively researched for over fifty years. Today, LI is a key part of many text processing pipelines, as text processing techniques generally assume that the language of the input text is known. Research in this area has recently been especially active. This article provides a brief history of LI research, and an extensive survey of the features and methods used so far in the LI literature. For describing the features and methods we introduce a unified notation. We discuss evaluation methods, applications of LI, as well as off-the-shelf LI systems that do not require training by the end user. Finally, we identify open issues, survey the work to date on each issue, and propose future directions for research in LI.",
keywords = "6121 Languages, 213 Electronic, automation and communications engineering, electronics",
author = "Tommi Jauhiainen and Marco Lui and Marcos Zampieri and Timothy Baldwin and Krister Lind{\'e}n",
note = "Under review at JAIR - Journal of Artificial Intelligence Research",
year = "2019",
month = "5",
day = "13",
language = "English",
journal = "Journal of Artificial Intelligence Research",
issn = "1076-9757",
publisher = "Association for the Advancement of Artificial Intelligence (AAAI)",

}

Automatic Language Identification in Texts : A Survey. / Jauhiainen, Tommi; Lui, Marco; Zampieri, Marcos; Baldwin, Timothy; Lindén, Krister.

In: Journal of Artificial Intelligence Research, 13.05.2019.

Research output: Contribution to journalArticleScientificpeer-review

TY - JOUR

T1 - Automatic Language Identification in Texts

T2 - A Survey

AU - Jauhiainen, Tommi

AU - Lui, Marco

AU - Zampieri, Marcos

AU - Baldwin, Timothy

AU - Lindén, Krister

N1 - Under review at JAIR - Journal of Artificial Intelligence Research

PY - 2019/5/13

Y1 - 2019/5/13

N2 - Language identification (LI) is the problem of determining the natural language that a document or part thereof is written in. Automatic LI has been extensively researched for over fifty years. Today, LI is a key part of many text processing pipelines, as text processing techniques generally assume that the language of the input text is known. Research in this area has recently been especially active. This article provides a brief history of LI research, and an extensive survey of the features and methods used so far in the LI literature. For describing the features and methods we introduce a unified notation. We discuss evaluation methods, applications of LI, as well as off-the-shelf LI systems that do not require training by the end user. Finally, we identify open issues, survey the work to date on each issue, and propose future directions for research in LI.

AB - Language identification (LI) is the problem of determining the natural language that a document or part thereof is written in. Automatic LI has been extensively researched for over fifty years. Today, LI is a key part of many text processing pipelines, as text processing techniques generally assume that the language of the input text is known. Research in this area has recently been especially active. This article provides a brief history of LI research, and an extensive survey of the features and methods used so far in the LI literature. For describing the features and methods we introduce a unified notation. We discuss evaluation methods, applications of LI, as well as off-the-shelf LI systems that do not require training by the end user. Finally, we identify open issues, survey the work to date on each issue, and propose future directions for research in LI.

KW - 6121 Languages

KW - 213 Electronic, automation and communications engineering, electronics

M3 - Article

JO - Journal of Artificial Intelligence Research

JF - Journal of Artificial Intelligence Research

SN - 1076-9757

ER -