OpusTools and Parallel Corpus Diagnostics

Mikko Aulamo; Umut Sulubacak; Sami Virpioja; Jörg Tiedemann

OpusTools and Parallel Corpus Diagnostics

Mikko Aulamo, Umut Sulubacak, Sami Virpioja, Jörg Tiedemann

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Scientific › peer-review

Abstract

This paper introduces OpusTools, a package for downloading and processing parallel corpora included in the OPUS corpus collection. The package implements tools for accessing compressed data in their archived release format and make it possible to easily convert between common formats. OpusTools also includes tools for language identification and data filtering as well as tools for importing data from various sources into the OPUS format. We show the use of these tools in parallel corpus creation and data diagnostics. The latter is especially useful for the identification of potential problems and errors in the extensive data set. Using these tools, we can now monitor the validity of data sets and improve the overall quality and consitency of the data collection.

Original language	English
Title of host publication	Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020)
Editors	Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, Stelios Piperidis
Number of pages	8
Place of Publication	Paris
Publisher	European Language Resources Association (ELRA)
Publication date	17 May 2020
Pages	3782-3789
ISBN (Electronic)	979-10-95546-34-4
Publication status	Published - 17 May 2020
MoE publication type	A4 Article in conference proceedings
Event	Language Resources and Evaluation Conference - [LREC 2020 was cancelled] Duration: 11 May 2020 → 16 May 2020 Conference number: 12 https://lrec2020.lrec-conf.org/

Bibliographical note

12th Edition of its Language Resources and Evaluation Conference was cancelled due to Covid 19 pandemic.

Fields of Science

113 Computer and information sciences
Machine translation
Corpus
Multilingual

Access to Document

2020.lrec-1.467Final published version, 654 KBLicence: CC BY-NC

FoTran: Found in Translation - Natural Language Understanding with Cross-Lingual Grounding
Tiedemann, J. (Project manager), Attieh, J. (Participant), Aulamo, M. (Participant), Celikkanat, H. (Participant), De Gibert Bonet, O. (Participant), Grönroos, S.-A. (Participant), Mickus, T. (Participant), Raganato, A. (Participant), Scherrer, Y. (Participant), Silfverberg, M. (Participant), Sjöblom, E. I. (Participant), Talman, A. (Participant), Vazquez , R. (Participant), Virpioja, S. P. (Participant), Yli-Jyrä, A. (Participant), Zosa, E. (Participant), Celikkanat, H. (Participant), Raganato, A. (Participant), Silfverberg, M. (Participant), Sulubacak, U. (Participant) & Vazquez , R. (Participant)
01/09/2018 → 31/03/2024
Project: EU Horizon 2020: European Research Council: Consolidator Grant (H2020-ERC-COG)
NLPL: Nordic Language Processing Laboratory
Tiedemann, J. (Participant) & Scherrer, Y. (Participant)
01/01/2017 → 31/12/2019
Project: Other project

OPUS
Tiedemann, J. (Creator), University of Helsinki, 2017
http://opus.nlpl.eu
Dataset

Cite this

Aulamo, M., Sulubacak, U., Virpioja, S., & Tiedemann, J. (2020). OpusTools and Parallel Corpus Diagnostics. In N. Calzolari, F. Béchet, P. Blache, K. Choukri, C. Cieri, T. Declerck, S. Goggi, H. Isahara, B. Maegaard, J. Mariani, H. Mazo, A. Moreno, J. Odijk, & S. Piperidis (Eds.), Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020) (pp. 3782-3789). European Language Resources Association (ELRA).

Aulamo, Mikko ; Sulubacak, Umut ; Virpioja, Sami et al. / OpusTools and Parallel Corpus Diagnostics. Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020). editor / Nicoletta Calzolari ; Frédéric Béchet ; Philippe Blache ; Khalid Choukri ; Christopher Cieri ; Thierry Declerck ; Sara Goggi ; Hitoshi Isahara ; Bente Maegaard ; Joseph Mariani ; Hélène Mazo ; Asuncion Moreno ; Jan Odijk ; Stelios Piperidis. Paris : European Language Resources Association (ELRA), 2020. pp. 3782-3789

@inproceedings{c220061f4556435bac85f8254ba182b7,

title = "OpusTools and Parallel Corpus Diagnostics",

abstract = "This paper introduces OpusTools, a package for downloading and processing parallel corpora included in the OPUS corpus collection. The package implements tools for accessing compressed data in their archived release format and make it possible to easily convert between common formats. OpusTools also includes tools for language identification and data filtering as well as tools for importing data from various sources into the OPUS format. We show the use of these tools in parallel corpus creation and data diagnostics. The latter is especially useful for the identification of potential problems and errors in the extensive data set. Using these tools, we can now monitor the validity of data sets and improve the overall quality and consitency of the data collection.",

keywords = "113 Computer and information sciences, Machine translation, Corpus, Multilingual",

author = "Mikko Aulamo and Umut Sulubacak and Sami Virpioja and J{\"o}rg Tiedemann",

note = "12th Edition of its Language Resources and Evaluation Conference was cancelled due to Covid 19 pandemic.; Language Resources and Evaluation Conference, LREC 2020 ; Conference date: 11-05-2020 Through 16-05-2020",

year = "2020",

month = may,

day = "17",

language = "English",

pages = "3782--3789",

editor = "Nicoletta Calzolari and Fr{\'e}d{\'e}ric B{\'e}chet and Philippe Blache and Khalid Choukri and Christopher Cieri and Thierry Declerck and Sara Goggi and Hitoshi Isahara and Bente Maegaard and Joseph Mariani and H{\'e}l{\`e}ne Mazo and Asuncion Moreno and Jan Odijk and Stelios Piperidis",

booktitle = "Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020)",

publisher = "European Language Resources Association (ELRA)",

address = "France",

url = "https://lrec2020.lrec-conf.org/",

}

Aulamo, M, Sulubacak, U, Virpioja, S & Tiedemann, J 2020, OpusTools and Parallel Corpus Diagnostics. in N Calzolari, F Béchet, P Blache, K Choukri, C Cieri, T Declerck, S Goggi, H Isahara, B Maegaard, J Mariani, H Mazo, A Moreno, J Odijk & S Piperidis (eds), Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020). European Language Resources Association (ELRA), Paris, pp. 3782-3789, Language Resources and Evaluation Conference, 11/05/2020.

OpusTools and Parallel Corpus Diagnostics. / Aulamo, Mikko; Sulubacak, Umut; Virpioja, Sami et al.
Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020). ed. / Nicoletta Calzolari; Frédéric Béchet; Philippe Blache; Khalid Choukri; Christopher Cieri; Thierry Declerck; Sara Goggi; Hitoshi Isahara; Bente Maegaard; Joseph Mariani; Hélène Mazo; Asuncion Moreno; Jan Odijk; Stelios Piperidis. Paris: European Language Resources Association (ELRA), 2020. p. 3782-3789.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Scientific › peer-review

TY - GEN

T1 - OpusTools and Parallel Corpus Diagnostics

AU - Aulamo, Mikko

AU - Sulubacak, Umut

AU - Virpioja, Sami

AU - Tiedemann, Jörg

N1 - Conference code: 12

PY - 2020/5/17

Y1 - 2020/5/17

N2 - This paper introduces OpusTools, a package for downloading and processing parallel corpora included in the OPUS corpus collection. The package implements tools for accessing compressed data in their archived release format and make it possible to easily convert between common formats. OpusTools also includes tools for language identification and data filtering as well as tools for importing data from various sources into the OPUS format. We show the use of these tools in parallel corpus creation and data diagnostics. The latter is especially useful for the identification of potential problems and errors in the extensive data set. Using these tools, we can now monitor the validity of data sets and improve the overall quality and consitency of the data collection.

AB - This paper introduces OpusTools, a package for downloading and processing parallel corpora included in the OPUS corpus collection. The package implements tools for accessing compressed data in their archived release format and make it possible to easily convert between common formats. OpusTools also includes tools for language identification and data filtering as well as tools for importing data from various sources into the OPUS format. We show the use of these tools in parallel corpus creation and data diagnostics. The latter is especially useful for the identification of potential problems and errors in the extensive data set. Using these tools, we can now monitor the validity of data sets and improve the overall quality and consitency of the data collection.

KW - 113 Computer and information sciences

KW - Machine translation

KW - Corpus

KW - Multilingual

M3 - Conference contribution

SP - 3782

EP - 3789

BT - Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020)

A2 - Calzolari, Nicoletta

A2 - Béchet, Frédéric

A2 - Blache, Philippe

A2 - Choukri, Khalid

A2 - Cieri, Christopher

A2 - Declerck, Thierry

A2 - Goggi, Sara

A2 - Isahara, Hitoshi

A2 - Maegaard, Bente

A2 - Mariani, Joseph

A2 - Mazo, Hélène

A2 - Moreno, Asuncion

A2 - Odijk, Jan

A2 - Piperidis, Stelios

PB - European Language Resources Association (ELRA)

CY - Paris

T2 - Language Resources and Evaluation Conference

Y2 - 11 May 2020 through 16 May 2020

ER -

Aulamo M, Sulubacak U, Virpioja S , Tiedemann J. OpusTools and Parallel Corpus Diagnostics. In Calzolari N, Béchet F, Blache P, Choukri K, Cieri C, Declerck T, Goggi S, Isahara H, Maegaard B, Mariani J, Mazo H, Moreno A, Odijk J, Piperidis S, editors, Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020). Paris: European Language Resources Association (ELRA). 2020. p. 3782-3789

OpusTools and Parallel Corpus Diagnostics

Abstract

Bibliographical note

Fields of Science

Access to Document

Projects

FoTran: Found in Translation - Natural Language Understanding with Cross-Lingual Grounding

NLPL: Nordic Language Processing Laboratory

Datasets

OPUS

Cite this