A New Massive Multilingual Dataset for High-Performance Language Technologies

Ona de Gibert; Graeme Nail; Nikolay Arefyev; Marta Bañón; Jelmer van der Linde; Shaoxiong Ji; Jaume Zaragoza-Bernabeu; Mikko Aulamo; Gema Ramírez-Sánchez; Andrey Kutuzov; Sampo Pyysalo; Stephan Oepen; Jörg Tiedemann

A New Massive Multilingual Dataset for High-Performance Language Technologies

Ona de Gibert, Graeme Nail, Nikolay Arefyev, Marta Bañón, Jelmer van der Linde, Shaoxiong Ji, Jaume Zaragoza-Bernabeu, Mikko Aulamo, Gema Ramírez-Sánchez, Andrey Kutuzov, Sampo Pyysalo, Stephan Oepen, Jörg Tiedemann

Tutkimustuotos: Artikkeli kirjassa/raportissa/konferenssijulkaisussa › Konferenssiartikkeli › Tieteellinen › vertaisarvioitu

Abstrakti

We present the HPLT (High Performance Language Technologies) language resources, a new massive multilingual dataset including both monolingual and bilingual corpora extracted from CommonCrawl and previously unused web crawls from the Internet Archive. We describe our methods for data acquisition, management and processing of large corpora, which rely on open-source software tools and high-performance computing. Our monolingual collection focuses on low- to medium-resourced languages and covers 75 languages and a total of ≈ 5.6 trillion word tokens de-duplicated on the document level. Our English-centric parallel corpus is derived from its monolingual counterpart and covers 18 language pairs and more than 96 million aligned sentence pairs with roughly 1.4 billion English tokens. The HPLT language resources are one of the largest open text corpora ever released, providing a great resource for language modeling and machine translation training. We publicly release the corpora, the software, and the tools used in this work.

Alkuperäiskieli	englanti
Otsikko	Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)
Toimittajat	Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, Nianwen Xue
Sivumäärä	13
Julkaisupaikka	Paris
Kustantaja	European Language Resources Association (ELRA)
Julkaisupäivä	2024
Sivut	1116-1128
ISBN (elektroninen)	978-2-493814-10-4
Tila	Julkaistu - 2024
OKM-julkaisutyyppi	A4 Artikkeli konferenssijulkaisuussa
Tapahtuma	The 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024) - Torino, Italia Kesto: 20 toukok. 2024 → 25 toukok. 2024

Julkaisusarja

Nimi	International conference on computational linguistics
Kustantaja	International Committee on Computational Linguistics
ISSN (painettu)	2951-2093

Nimi	LREC proceedings
Kustantaja	Language Resources Association (ELRA)
ISSN (elektroninen)	2522-2686

Lisätietoja

Publisher Copyright:
© 2024 ELRA Language Resource Association: CC BY-NC 4.0.

Tieteenalat

6121 Kielitieteet
113 Tietojenkäsittely- ja informaatiotieteet

Pääsy asiakirjaan

2024.lrec-main.100Lopullinen julkaistu versio, 1,23 MBLisenssi: CC BY-NC

Siteeraa tätä

de Gibert, O., Nail, G., Arefyev, N., Bañón, M., van der Linde, J., Ji, S., Zaragoza-Bernabeu, J., Aulamo, M., Ramírez-Sánchez, G., Kutuzov, A., Pyysalo, S., Oepen, S., & Tiedemann, J. (2024). A New Massive Multilingual Dataset for High-Performance Language Technologies. teoksessa N. Calzolari, M.-Y. Kan, V. Hoste, A. Lenci, S. Sakti, & N. Xue (Toimittajat), Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024) (Sivut 1116-1128). (International conference on computational linguistics ), (LREC proceedings ). European Language Resources Association (ELRA).

de Gibert, Ona ; Nail, Graeme ; Arefyev, Nikolay et al. / A New Massive Multilingual Dataset for High-Performance Language Technologies. Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024). Toimittaja / Nicoletta Calzolari ; Min-Yen Kan ; Veronique Hoste ; Alessandro Lenci ; Sakriani Sakti ; Nianwen Xue. Paris : European Language Resources Association (ELRA), 2024. Sivut 1116-1128 (International conference on computational linguistics ). (LREC proceedings ).

@inproceedings{f2f32c8b1aea4bfc89be866d8e3e803c,

title = "A New Massive Multilingual Dataset for High-Performance Language Technologies",

abstract = "We present the HPLT (High Performance Language Technologies) language resources, a new massive multilingual dataset including both monolingual and bilingual corpora extracted from CommonCrawl and previously unused web crawls from the Internet Archive. We describe our methods for data acquisition, management and processing of large corpora, which rely on open-source software tools and high-performance computing. Our monolingual collection focuses on low- to medium-resourced languages and covers 75 languages and a total of ≈ 5.6 trillion word tokens de-duplicated on the document level. Our English-centric parallel corpus is derived from its monolingual counterpart and covers 18 language pairs and more than 96 million aligned sentence pairs with roughly 1.4 billion English tokens. The HPLT language resources are one of the largest open text corpora ever released, providing a great resource for language modeling and machine translation training. We publicly release the corpora, the software, and the tools used in this work.",

keywords = "6121 Languages, 113 Computer and information sciences, Low-resource Languages, Monolingual Corpus, Parallel Corpus, Pre-training Datasets",

author = "{de Gibert}, Ona and Graeme Nail and Nikolay Arefyev and Marta Ba{\~n}{\'o}n and {van der Linde}, Jelmer and Shaoxiong Ji and Jaume Zaragoza-Bernabeu and Mikko Aulamo and Gema Ram{\'i}rez-S{\'a}nchez and Andrey Kutuzov and Sampo Pyysalo and Stephan Oepen and J{\"o}rg Tiedemann",

note = "Publisher Copyright: {\textcopyright} 2024 ELRA Language Resource Association: CC BY-NC 4.0.; Joint International Conference on Computational Linguistics, Language Resources and Evaluation, LREC-COLING 2024 ; Conference date: 20-05-2024 Through 25-05-2024",

year = "2024",

language = "English",

series = "International conference on computational linguistics ",

publisher = "European Language Resources Association (ELRA)",

pages = "1116--1128",

editor = "Nicoletta Calzolari and Min-Yen Kan and Veronique Hoste and Alessandro Lenci and Sakriani Sakti and Nianwen Xue",

booktitle = "Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)",

address = "France",

}

de Gibert, O, Nail, G, Arefyev, N, Bañón, M, van der Linde, J, Ji, S, Zaragoza-Bernabeu, J, Aulamo, M, Ramírez-Sánchez, G, Kutuzov, A, Pyysalo, S, Oepen, S & Tiedemann, J 2024, A New Massive Multilingual Dataset for High-Performance Language Technologies. julkaisussa N Calzolari, M-Y Kan, V Hoste, A Lenci, S Sakti & N Xue (toim), Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024). International conference on computational linguistics , LREC proceedings , European Language Resources Association (ELRA), Paris, Sivut 1116-1128, The 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), Torino, Italia, 20/05/2024.

A New Massive Multilingual Dataset for High-Performance Language Technologies. / de Gibert, Ona; Nail, Graeme; Arefyev, Nikolay et al.
Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024). toim. / Nicoletta Calzolari; Min-Yen Kan; Veronique Hoste; Alessandro Lenci; Sakriani Sakti; Nianwen Xue. Paris: European Language Resources Association (ELRA), 2024. s. 1116-1128 (International conference on computational linguistics ), (LREC proceedings ).

Tutkimustuotos: Artikkeli kirjassa/raportissa/konferenssijulkaisussa › Konferenssiartikkeli › Tieteellinen › vertaisarvioitu

TY - GEN

T1 - A New Massive Multilingual Dataset for High-Performance Language Technologies

AU - de Gibert, Ona

AU - Nail, Graeme

AU - Arefyev, Nikolay

AU - Bañón, Marta

AU - van der Linde, Jelmer

AU - Ji, Shaoxiong

AU - Zaragoza-Bernabeu, Jaume

AU - Aulamo, Mikko

AU - Ramírez-Sánchez, Gema

AU - Kutuzov, Andrey

AU - Pyysalo, Sampo

AU - Oepen, Stephan

AU - Tiedemann, Jörg

PY - 2024

Y1 - 2024

N2 - We present the HPLT (High Performance Language Technologies) language resources, a new massive multilingual dataset including both monolingual and bilingual corpora extracted from CommonCrawl and previously unused web crawls from the Internet Archive. We describe our methods for data acquisition, management and processing of large corpora, which rely on open-source software tools and high-performance computing. Our monolingual collection focuses on low- to medium-resourced languages and covers 75 languages and a total of ≈ 5.6 trillion word tokens de-duplicated on the document level. Our English-centric parallel corpus is derived from its monolingual counterpart and covers 18 language pairs and more than 96 million aligned sentence pairs with roughly 1.4 billion English tokens. The HPLT language resources are one of the largest open text corpora ever released, providing a great resource for language modeling and machine translation training. We publicly release the corpora, the software, and the tools used in this work.

AB - We present the HPLT (High Performance Language Technologies) language resources, a new massive multilingual dataset including both monolingual and bilingual corpora extracted from CommonCrawl and previously unused web crawls from the Internet Archive. We describe our methods for data acquisition, management and processing of large corpora, which rely on open-source software tools and high-performance computing. Our monolingual collection focuses on low- to medium-resourced languages and covers 75 languages and a total of ≈ 5.6 trillion word tokens de-duplicated on the document level. Our English-centric parallel corpus is derived from its monolingual counterpart and covers 18 language pairs and more than 96 million aligned sentence pairs with roughly 1.4 billion English tokens. The HPLT language resources are one of the largest open text corpora ever released, providing a great resource for language modeling and machine translation training. We publicly release the corpora, the software, and the tools used in this work.

KW - 6121 Languages

KW - 113 Computer and information sciences

KW - Low-resource Languages

KW - Monolingual Corpus

KW - Parallel Corpus

KW - Pre-training Datasets

M3 - Conference contribution

AN - SCOPUS:85195967823

T3 - International conference on computational linguistics

SP - 1116

EP - 1128

BT - Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)

A2 - Calzolari, Nicoletta

A2 - Kan, Min-Yen

A2 - Hoste, Veronique

A2 - Lenci, Alessandro

A2 - Sakti, Sakriani

A2 - Xue, Nianwen

PB - European Language Resources Association (ELRA)

CY - Paris

T2 - Joint International Conference on Computational Linguistics, Language Resources and Evaluation

Y2 - 20 May 2024 through 25 May 2024

ER -

de Gibert O, Nail G, Arefyev N, Bañón M, van der Linde J, Ji S et al. A New Massive Multilingual Dataset for High-Performance Language Technologies. julkaisussa Calzolari N, Kan MY, Hoste V, Lenci A, Sakti S, Xue N, toimittajat, Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024). Paris: European Language Resources Association (ELRA). 2024. s. 1116-1128. (International conference on computational linguistics ). (LREC proceedings ).