The OPUS Resource Repository: An Open Package for Creating Parallel Corpora and Machine Translation Services

Mikko Aulamo; Jörg Tiedemann

The OPUS Resource Repository: An Open Package for Creating Parallel Corpora and Machine Translation Services

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Scientific › peer-review

Abstract

This paper presents a flexible and powerful system for creating parallel corpora and for running neural machine translation services. Our package provides a scalable data repository backend that offers transparent data pre-processing pipelines and automatic alignment procedures that facilitate the compilation of extensive parallel data sets from a variety of sources. Moreover, we develop a web-based interface that constitutes an intuitive frontend for end-users of the platform. The whole system can easily be distributed over virtual machines and implements a sophisticated permission system with secure connections and a flexible database for storing arbitrary metadata. Furthermore, we also provide an interface for neural machine translation that can run as a service on virtual machines, which also incorporates a connection to the data repository software.

Original language	English
Title of host publication	22nd Nordic Conference on Computational Linguistics (NoDaLiDa) : Proceedings of the Conference
Editors	Mareike Hartmann, Barbara Plank
Number of pages	6
Place of Publication	Linköping
Publisher	Linköping University Electronic Press
Publication date	2019
Pages	389–394
ISBN (Electronic)	978-91-7929-995-8
Publication status	Published - 2019
MoE publication type	A4 Article in conference proceedings
Event	Nordic Conference on Computational Linguistics - Turku, Finland Duration: 30 Sept 2019 → 2 Oct 2019 Conference number: 22 https://nodalida2019.org/

Publication series

Name	Linköping Electronic Conference Proceedings
Publisher	Linköping University Electronic Press
Number	167
ISSN (Print)	1650-3686
ISSN (Electronic)	1650-3740

Name	NEALT Proceedings Series
Number	42

Fields of Science

113 Computer and information sciences
6121 Languages

Access to Document

W19-6146Final published version, 603 KBLicence: CC BY

Cite this

Aulamo, M., & Tiedemann, J. (2019). The OPUS Resource Repository: An Open Package for Creating Parallel Corpora and Machine Translation Services. In M. Hartmann, & B. Plank (Eds.), 22nd Nordic Conference on Computational Linguistics (NoDaLiDa): Proceedings of the Conference (pp. 389–394). (Linköping Electronic Conference Proceedings; No. 167), (NEALT Proceedings Series; No. 42). Linköping University Electronic Press.

Aulamo, Mikko ; Tiedemann, Jörg. / The OPUS Resource Repository : An Open Package for Creating Parallel Corpora and Machine Translation Services. 22nd Nordic Conference on Computational Linguistics (NoDaLiDa): Proceedings of the Conference. editor / Mareike Hartmann ; Barbara Plank. Linköping : Linköping University Electronic Press, 2019. pp. 389–394 (Linköping Electronic Conference Proceedings; 167). (NEALT Proceedings Series; 42).

@inproceedings{e73384c4b7314426a17bd44e8066bbf3,

title = "The OPUS Resource Repository: An Open Package for Creating Parallel Corpora and Machine Translation Services",

abstract = "This paper presents a flexible and powerful system for creating parallel corpora and for running neural machine translation services. Our package provides a scalable data repository backend that offers transparent data pre-processing pipelines and automatic alignment procedures that facilitate the compilation of extensive parallel data sets from a variety of sources. Moreover, we develop a web-based interface that constitutes an intuitive frontend for end-users of the platform. The whole system can easily be distributed over virtual machines and implements a sophisticated permission system with secure connections and a flexible database for storing arbitrary metadata. Furthermore, we also provide an interface for neural machine translation that can run as a service on virtual machines, which also incorporates a connection to the data repository software.",

keywords = "113 Computer and information sciences, 6121 Languages",

author = "Mikko Aulamo and J{\"o}rg Tiedemann",

year = "2019",

language = "English",

series = "Link{\"o}ping Electronic Conference Proceedings",

publisher = "Link{\"o}ping University Electronic Press",

number = "167",

pages = "389–394",

editor = "Mareike Hartmann and Barbara Plank",

booktitle = "22nd Nordic Conference on Computational Linguistics (NoDaLiDa)",

address = "Sweden",

note = "Nordic Conference on Computational Linguistics, NoDaLiDa ; Conference date: 30-09-2019 Through 02-10-2019",

url = "https://nodalida2019.org/",

}

Aulamo, M & Tiedemann, J 2019, The OPUS Resource Repository: An Open Package for Creating Parallel Corpora and Machine Translation Services. in M Hartmann & B Plank (eds), 22nd Nordic Conference on Computational Linguistics (NoDaLiDa): Proceedings of the Conference. Linköping Electronic Conference Proceedings, no. 167, NEALT Proceedings Series, no. 42, Linköping University Electronic Press, Linköping, pp. 389–394, Nordic Conference on Computational Linguistics, Turku, Finland, 30/09/2019.

The OPUS Resource Repository: An Open Package for Creating Parallel Corpora and Machine Translation Services. / Aulamo, Mikko ; Tiedemann, Jörg.
22nd Nordic Conference on Computational Linguistics (NoDaLiDa): Proceedings of the Conference. ed. / Mareike Hartmann; Barbara Plank. Linköping: Linköping University Electronic Press, 2019. p. 389–394 (Linköping Electronic Conference Proceedings; No. 167), (NEALT Proceedings Series; No. 42).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Scientific › peer-review

TY - GEN

T1 - The OPUS Resource Repository

T2 - Nordic Conference on Computational Linguistics

AU - Aulamo, Mikko

AU - Tiedemann, Jörg

N1 - Conference code: 22

PY - 2019

Y1 - 2019

N2 - This paper presents a flexible and powerful system for creating parallel corpora and for running neural machine translation services. Our package provides a scalable data repository backend that offers transparent data pre-processing pipelines and automatic alignment procedures that facilitate the compilation of extensive parallel data sets from a variety of sources. Moreover, we develop a web-based interface that constitutes an intuitive frontend for end-users of the platform. The whole system can easily be distributed over virtual machines and implements a sophisticated permission system with secure connections and a flexible database for storing arbitrary metadata. Furthermore, we also provide an interface for neural machine translation that can run as a service on virtual machines, which also incorporates a connection to the data repository software.

AB - This paper presents a flexible and powerful system for creating parallel corpora and for running neural machine translation services. Our package provides a scalable data repository backend that offers transparent data pre-processing pipelines and automatic alignment procedures that facilitate the compilation of extensive parallel data sets from a variety of sources. Moreover, we develop a web-based interface that constitutes an intuitive frontend for end-users of the platform. The whole system can easily be distributed over virtual machines and implements a sophisticated permission system with secure connections and a flexible database for storing arbitrary metadata. Furthermore, we also provide an interface for neural machine translation that can run as a service on virtual machines, which also incorporates a connection to the data repository software.

KW - 113 Computer and information sciences

KW - 6121 Languages

M3 - Conference contribution

T3 - Linköping Electronic Conference Proceedings

SP - 389

EP - 394

BT - 22nd Nordic Conference on Computational Linguistics (NoDaLiDa)

A2 - Hartmann, Mareike

A2 - Plank, Barbara

PB - Linköping University Electronic Press

CY - Linköping

Y2 - 30 September 2019 through 2 October 2019

ER -

Aulamo M , Tiedemann J. The OPUS Resource Repository: An Open Package for Creating Parallel Corpora and Machine Translation Services. In Hartmann M, Plank B, editors, 22nd Nordic Conference on Computational Linguistics (NoDaLiDa): Proceedings of the Conference. Linköping: Linköping University Electronic Press. 2019. p. 389–394. (Linköping Electronic Conference Proceedings; 167). (NEALT Proceedings Series; 42).