On the differences between BERT and MT encoder spaces and how to address them in translation tasks

Raul Vazquez; Hande Celikkanat; Mathias Creutz; Jörg Tiedemann

doi:10.18653/v1/2021.acl-srw.35

On the differences between BERT and MT encoder spaces and how to address them in translation tasks

Raul Vazquez , Hande Celikkanat, Mathias Creutz, Jörg Tiedemann

Tutkimustuotos: Artikkeli kirjassa/raportissa/konferenssijulkaisussa › Konferenssiartikkeli › Tieteellinen › vertaisarvioitu

Abstrakti

Various studies show that pretrained language models such as BERT cannot straightforwardly replace encoders in neural machine translation despite their enormous success in other tasks. This is even more astonishing considering the similarities between the architectures. This paper sheds some light on the embedding spaces they create, using average cosine similarity, contextuality metrics and measures for representational similarity for comparison, revealing that BERT and NMT encoder representations look significantly different from one another. In order to address this issue, we propose a supervised transformation from one into the other using explicit alignment and fine-tuning. Our results demonstrate the need for such a transformation to improve the applicability of BERT in MT.

Alkuperäiskieli	englanti
Otsikko	Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing : Student Research Workshop
Toimittajat	Jad Kabbara, Haitao Lin, Amandalynne Paullada, Jannis Vamvas
Sivumäärä	11
Julkaisupaikka	Stroudsburg
Kustantaja	The Association for Computational Linguistics
Julkaisupäivä	elok. 2021
Sivut	337-347
ISBN (painettu)	978-1-954085-55-8
DOI - pysyväislinkit	https://doi.org/10.18653/v1/2021.acl-srw.35
Tila	Julkaistu - elok. 2021
OKM-julkaisutyyppi	A4 Artikkeli konferenssijulkaisuussa
Tapahtuma	The Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL-IJCNLP 2021) - Bangkok [Online event] Kesto: 5 elok. 2021 → 6 elok. 2021 Konferenssinumero: 59/11

Tieteenalat

113 Tietojenkäsittely- ja informaatiotieteet
6121 Kielitieteet

Pääsy asiakirjaan

10.18653/v1/2021.acl-srw.35Lisenssi: CC BY

2021.acl-srw.35Lopullinen julkaistu versio, 0,99 MBLisenssi: CC BY

FoTran: Found in Translation - Natural Language Understanding with Cross-Lingual Grounding
Tiedemann, J., Attieh, J., Aulamo, M., Boggia, M., Celikkanat, H., De Gibert Bonet, O., Grönroos, S., Mickus, T., Raganato, A., Scherrer, Y., Silfverberg, M., Sjöblom, E. I., Talman, A., Vazquez , R., Virpioja, S. P., Yli-Jyrä, A., Zosa, E., Celikkanat, H., Raganato, A., Silfverberg, M., Sulubacak, U. & Vazquez , R.
01/09/2018 → 31/03/2024
Projekti: EU Horizon 2020: European Research Council: Consolidator Grant (H2020-ERC-COG)

Siteeraa tätä

Vazquez , R., Celikkanat, H., Creutz, M., & Tiedemann, J. (2021). On the differences between BERT and MT encoder spaces and how to address them in translation tasks. teoksessa J. Kabbara, H. Lin, A. Paullada, & J. Vamvas (Toimittajat), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Student Research Workshop (Sivut 337-347). The Association for Computational Linguistics. https://doi.org/10.18653/v1/2021.acl-srw.35

Vazquez , Raul ; Celikkanat, Hande ; Creutz, Mathias et al. / On the differences between BERT and MT encoder spaces and how to address them in translation tasks. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Student Research Workshop. Toimittaja / Jad Kabbara ; Haitao Lin ; Amandalynne Paullada ; Jannis Vamvas. Stroudsburg : The Association for Computational Linguistics, 2021. Sivut 337-347

@inproceedings{1d9dd4dca38e4631869a4f0901397bac,

title = "On the differences between BERT and MT encoder spaces and how to address them in translation tasks",

abstract = "Various studies show that pretrained language models such as BERT cannot straightforwardly replace encoders in neural machine translation despite their enormous success in other tasks. This is even more astonishing considering the similarities between the architectures. This paper sheds some light on the embedding spaces they create, using average cosine similarity, contextuality metrics and measures for representational similarity for comparison, revealing that BERT and NMT encoder representations look significantly different from one another. In order to address this issue, we propose a supervised transformation from one into the other using explicit alignment and fine-tuning. Our results demonstrate the need for such a transformation to improve the applicability of BERT in MT.",

keywords = "113 Computer and information sciences, 6121 Languages",

author = "Raul Vazquez and Hande Celikkanat and Mathias Creutz and J{\"o}rg Tiedemann",

year = "2021",

month = aug,

doi = "10.18653/v1/2021.acl-srw.35",

language = "English",

isbn = "978-1-954085-55-8",

pages = "337--347",

editor = "Jad Kabbara and Lin, {Haitao } and Paullada, {Amandalynne } and Vamvas, {Jannis }",

booktitle = "Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing",

publisher = "The Association for Computational Linguistics",

address = "United States",

note = "Annual Meeting of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021 ; Conference date: 05-08-2021 Through 06-08-2021",

}

Vazquez , R , Celikkanat, H , Creutz, M & Tiedemann, J 2021, On the differences between BERT and MT encoder spaces and how to address them in translation tasks. julkaisussa J Kabbara, H Lin, A Paullada & J Vamvas (toim), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Student Research Workshop. The Association for Computational Linguistics, Stroudsburg, Sivut 337-347, The Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL-IJCNLP 2021), 05/08/2021. https://doi.org/10.18653/v1/2021.acl-srw.35

On the differences between BERT and MT encoder spaces and how to address them in translation tasks. / Vazquez , Raul ; Celikkanat, Hande ; Creutz, Mathias et al.
Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Student Research Workshop. toim. / Jad Kabbara; Haitao Lin; Amandalynne Paullada; Jannis Vamvas. Stroudsburg: The Association for Computational Linguistics, 2021. s. 337-347.

Tutkimustuotos: Artikkeli kirjassa/raportissa/konferenssijulkaisussa › Konferenssiartikkeli › Tieteellinen › vertaisarvioitu

TY - GEN

T1 - On the differences between BERT and MT encoder spaces and how to address them in translation tasks

AU - Vazquez , Raul

AU - Celikkanat, Hande

AU - Creutz, Mathias

AU - Tiedemann, Jörg

N1 - Conference code: 59/11

PY - 2021/8

Y1 - 2021/8

N2 - Various studies show that pretrained language models such as BERT cannot straightforwardly replace encoders in neural machine translation despite their enormous success in other tasks. This is even more astonishing considering the similarities between the architectures. This paper sheds some light on the embedding spaces they create, using average cosine similarity, contextuality metrics and measures for representational similarity for comparison, revealing that BERT and NMT encoder representations look significantly different from one another. In order to address this issue, we propose a supervised transformation from one into the other using explicit alignment and fine-tuning. Our results demonstrate the need for such a transformation to improve the applicability of BERT in MT.

AB - Various studies show that pretrained language models such as BERT cannot straightforwardly replace encoders in neural machine translation despite their enormous success in other tasks. This is even more astonishing considering the similarities between the architectures. This paper sheds some light on the embedding spaces they create, using average cosine similarity, contextuality metrics and measures for representational similarity for comparison, revealing that BERT and NMT encoder representations look significantly different from one another. In order to address this issue, we propose a supervised transformation from one into the other using explicit alignment and fine-tuning. Our results demonstrate the need for such a transformation to improve the applicability of BERT in MT.

KW - 113 Computer and information sciences

KW - 6121 Languages

U2 - 10.18653/v1/2021.acl-srw.35

DO - 10.18653/v1/2021.acl-srw.35

M3 - Conference contribution

SN - 978-1-954085-55-8

SP - 337

EP - 347

BT - Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing

A2 - Kabbara, Jad

A2 - Lin, Haitao

A2 - Paullada, Amandalynne

A2 - Vamvas, Jannis

PB - The Association for Computational Linguistics

CY - Stroudsburg

T2 - Annual Meeting of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing

Y2 - 5 August 2021 through 6 August 2021

ER -

Vazquez R , Celikkanat H , Creutz M , Tiedemann J. On the differences between BERT and MT encoder spaces and how to address them in translation tasks. julkaisussa Kabbara J, Lin H, Paullada A, Vamvas J, toimittajat, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Student Research Workshop. Stroudsburg: The Association for Computational Linguistics. 2021. s. 337-347 doi: 10.18653/v1/2021.acl-srw.35

On the differences between BERT and MT encoder spaces and how to address them in translation tasks

Abstrakti

Tieteenalat

Pääsy asiakirjaan

Projektit

FoTran: Found in Translation - Natural Language Understanding with Cross-Lingual Grounding

Siteeraa tätä