Murreviikko - A Dialectologically Annotated and Normalized Dataset of Finnish Tweets

Olli Vilhelm Kuparinen

Murreviikko - A Dialectologically Annotated and Normalized Dataset of Finnish Tweets

Olli Vilhelm Kuparinen

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Scientific › peer-review

Abstract

This paper presents Murreviikko, a dataset of dialectal Finnish tweets which have been dialectologically annotated and manually normalized to a standard form. The dataset can be used as a test set for dialect identification and dialect-to-standard normalization, for instance. We evaluate the dataset on the normalization task, comparing an existing normalization model built on a spoken dialect corpus and three newly trained models with different architectures. We find that there are significant differences in normalization difficulty between the dialects, and that a character-level statistical machine translation model performs best on the Murreviikko tweet dataset.

Original language	English
Title of host publication	Tenth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2023) : Proceedings of the Workshop
Editors	Yves Scherrer, Tommi Jauhiainen, Nikola Ljubešić, Preslav Nakov, Jörg Tiedemann, Marcos Zampieri
Number of pages	9
Place of Publication	Stroudsburg
Publisher	The Association for Computational Linguistics
Publication date	5 May 2023
Pages	31-39
ISBN (Electronic)	978-1-959429-50-0
Publication status	Published - 5 May 2023
MoE publication type	A4 Article in conference proceedings
Event	Workshop on NLP for Similar Languages, Varieties and Dialects - Dubrovnik, Croatia Duration: 5 May 2023 → 6 May 2023 Conference number: 10 https://sites.google.com/view/vardial-2023

Fields of Science

6121 Languages
113 Computer and information sciences

Access to Document

2023.vardial-1.3Final published version, 769 KBLicence: CC BY

https://aclanthology.org/2023.vardial-1.3Licence: CC BY

CorCoDial: Corpus-based computational dialectology: exploiting machine translation techniques to extract, visualize and interpret dialectal patterns
Scherrer, Y. (Project manager), Tiedemann, J. (Project manager), Mickus, T. (Participant), Miletic Haddad, A. (Participant), Psaltaki, E. (Participant), Roemling, D. (Participant), Siewert, J. (Participant) & Siewert, J. (Participant)
Suomen Akatemia Projektilaskutus
01/09/2021 → 31/08/2025
Project: Research Council of Finland: Academy Project

Murreviikko: an Annotated and Normalized Corpus of Dialectal Finnish Tweets
Kuparinen, O. V. (Creator), Zenodo, 2023
DOI: 10.5281/zenodo.7907535
Dataset

Cite this

Kuparinen, O. V. (2023). Murreviikko - A Dialectologically Annotated and Normalized Dataset of Finnish Tweets. In Y. Scherrer, T. Jauhiainen, N. Ljubešić, P. Nakov, J. Tiedemann, & M. Zampieri (Eds.), Tenth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2023): Proceedings of the Workshop (pp. 31-39). The Association for Computational Linguistics. https://aclanthology.org/2023.vardial-1.3

Kuparinen, Olli Vilhelm. / Murreviikko - A Dialectologically Annotated and Normalized Dataset of Finnish Tweets. Tenth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2023): Proceedings of the Workshop. editor / Yves Scherrer ; Tommi Jauhiainen ; Nikola Ljubešić ; Preslav Nakov ; Jörg Tiedemann ; Marcos Zampieri. Stroudsburg : The Association for Computational Linguistics, 2023. pp. 31-39

@inproceedings{3d948118c03c401298b7ebffb208bcb1,

title = "Murreviikko - A Dialectologically Annotated and Normalized Dataset of Finnish Tweets",

abstract = "This paper presents Murreviikko, a dataset of dialectal Finnish tweets which have been dialectologically annotated and manually normalized to a standard form. The dataset can be used as a test set for dialect identification and dialect-to-standard normalization, for instance. We evaluate the dataset on the normalization task, comparing an existing normalization model built on a spoken dialect corpus and three newly trained models with different architectures. We find that there are significant differences in normalization difficulty between the dialects, and that a character-level statistical machine translation model performs best on the Murreviikko tweet dataset.",

keywords = "6121 Languages, 113 Computer and information sciences",

author = "Kuparinen, {Olli Vilhelm}",

year = "2023",

month = may,

day = "5",

language = "English",

pages = "31--39",

editor = "Yves Scherrer and Jauhiainen, {Tommi } and Ljube{\v s}i{\'c}, {Nikola } and Nakov, {Preslav } and J{\"o}rg Tiedemann and Zampieri, {Marcos }",

booktitle = "Tenth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2023)",

publisher = "The Association for Computational Linguistics",

address = "United States",

note = "Workshop on NLP for Similar Languages, Varieties and Dialects, VarDial 2023 ; Conference date: 05-05-2023 Through 06-05-2023",

url = "https://sites.google.com/view/vardial-2023",

}

Kuparinen, OV 2023, Murreviikko - A Dialectologically Annotated and Normalized Dataset of Finnish Tweets. in Y Scherrer, T Jauhiainen, N Ljubešić, P Nakov, J Tiedemann & M Zampieri (eds), Tenth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2023): Proceedings of the Workshop. The Association for Computational Linguistics, Stroudsburg, pp. 31-39, Workshop on NLP for Similar Languages, Varieties and Dialects, Dubrovnik, Croatia, 05/05/2023. <https://aclanthology.org/2023.vardial-1.3>

Murreviikko - A Dialectologically Annotated and Normalized Dataset of Finnish Tweets. / Kuparinen, Olli Vilhelm.
Tenth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2023): Proceedings of the Workshop. ed. / Yves Scherrer; Tommi Jauhiainen; Nikola Ljubešić; Preslav Nakov; Jörg Tiedemann; Marcos Zampieri. Stroudsburg: The Association for Computational Linguistics, 2023. p. 31-39.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Scientific › peer-review

TY - GEN

T1 - Murreviikko - A Dialectologically Annotated and Normalized Dataset of Finnish Tweets

AU - Kuparinen, Olli Vilhelm

N1 - Conference code: 10

PY - 2023/5/5

Y1 - 2023/5/5

N2 - This paper presents Murreviikko, a dataset of dialectal Finnish tweets which have been dialectologically annotated and manually normalized to a standard form. The dataset can be used as a test set for dialect identification and dialect-to-standard normalization, for instance. We evaluate the dataset on the normalization task, comparing an existing normalization model built on a spoken dialect corpus and three newly trained models with different architectures. We find that there are significant differences in normalization difficulty between the dialects, and that a character-level statistical machine translation model performs best on the Murreviikko tweet dataset.

AB - This paper presents Murreviikko, a dataset of dialectal Finnish tweets which have been dialectologically annotated and manually normalized to a standard form. The dataset can be used as a test set for dialect identification and dialect-to-standard normalization, for instance. We evaluate the dataset on the normalization task, comparing an existing normalization model built on a spoken dialect corpus and three newly trained models with different architectures. We find that there are significant differences in normalization difficulty between the dialects, and that a character-level statistical machine translation model performs best on the Murreviikko tweet dataset.

KW - 6121 Languages

KW - 113 Computer and information sciences

M3 - Conference contribution

SP - 31

EP - 39

BT - Tenth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2023)

A2 - Scherrer, Yves

A2 - Jauhiainen, Tommi

A2 - Ljubešić, Nikola

A2 - Nakov, Preslav

A2 - Tiedemann, Jörg

A2 - Zampieri, Marcos

PB - The Association for Computational Linguistics

CY - Stroudsburg

T2 - Workshop on NLP for Similar Languages, Varieties and Dialects

Y2 - 5 May 2023 through 6 May 2023

ER -

Kuparinen OV. Murreviikko - A Dialectologically Annotated and Normalized Dataset of Finnish Tweets. In Scherrer Y, Jauhiainen T, Ljubešić N, Nakov P, Tiedemann J, Zampieri M, editors, Tenth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2023): Proceedings of the Workshop. Stroudsburg: The Association for Computational Linguistics. 2023. p. 31-39

Murreviikko - A Dialectologically Annotated and Normalized Dataset of Finnish Tweets

Abstract

Fields of Science

Access to Document

Projects

CorCoDial: Corpus-based computational dialectology: exploiting machine translation techniques to extract, visualize and interpret dialectal patterns

Datasets

Murreviikko: an Annotated and Normalized Corpus of Dialectal Finnish Tweets

Cite this