On the differences between BERT and MT encoder spaces and how to address them in translation tasks

Raul Vazquez; Hande Celikkanat; Mathias Creutz; Jörg Tiedemann

doi:10.18653/v1/2021.acl-srw.35

On the differences between BERT and MT encoder spaces and how to address them in translation tasks

Raul Vazquez , Hande Celikkanat, Mathias Creutz, Jörg Tiedemann

Forskningsoutput: Kapitel i bok/rapport/konferenshandling › Konferensbidrag › Vetenskaplig › Peer review

Sammanfattning

Various studies show that pretrained language models such as BERT cannot straightforwardly replace encoders in neural machine translation despite their enormous success in other tasks. This is even more astonishing considering the similarities between the architectures. This paper sheds some light on the embedding spaces they create, using average cosine similarity, contextuality metrics and measures for representational similarity for comparison, revealing that BERT and NMT encoder representations look significantly different from one another. In order to address this issue, we propose a supervised transformation from one into the other using explicit alignment and fine-tuning. Our results demonstrate the need for such a transformation to improve the applicability of BERT in MT.

Originalspråk	engelska
Titel på värdpublikation	Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing : Student Research Workshop
Redaktörer	Jad Kabbara, Haitao Lin, Amandalynne Paullada, Jannis Vamvas
Antal sidor	11
Utgivningsort	Stroudsburg
Förlag	The Association for Computational Linguistics
Utgivningsdatum	aug. 2021
Sidor	337-347
ISBN (tryckt)	978-1-954085-55-8
DOI	https://doi.org/10.18653/v1/2021.acl-srw.35
Status	Publicerad - aug. 2021
MoE-publikationstyp	A4 Artikel i en konferenspublikation
Evenemang	Annual Meeting of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing - Bangkok [Online event] Varaktighet: 5 aug. 2021 → 6 aug. 2021 Konferensnummer: 59/11

Vetenskapsgrenar

113 Data- och informationsvetenskap
6121 Språkvetenskaper

Åtkomst av dokument

10.18653/v1/2021.acl-srw.35Licens: CC BY

2021.acl-srw.35Slutlig publicerad version, 0,99 MBLicens: CC BY

FoTran: Found in Translation - Natural Language Understanding with Cross-Lingual Grounding
Tiedemann, J., Attieh, J., Aulamo, M., Boggia, M., Celikkanat, H., De Gibert Bonet, O., Grönroos, S., Mickus, T., Raganato, A., Scherrer, Y., Silfverberg, M., Sjöblom, E. I., Talman, A., Vazquez , R., Virpioja, S. P., Yli-Jyrä, A., Zosa, E., Celikkanat, H., Raganato, A., Silfverberg, M., Sulubacak, U. & Vazquez , R.
01/09/2018 → 31/03/2024
Projekt: EU Horizon 2020: European Research Council: Consolidator Grant (H2020-ERC-COG)

Citera det här

Vazquez , R., Celikkanat, H., Creutz, M., & Tiedemann, J. (2021). On the differences between BERT and MT encoder spaces and how to address them in translation tasks. I J. Kabbara, H. Lin, A. Paullada, & J. Vamvas (Red.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Student Research Workshop (s. 337-347). The Association for Computational Linguistics. https://doi.org/10.18653/v1/2021.acl-srw.35

Vazquez , Raul ; Celikkanat, Hande ; Creutz, Mathias et al. / On the differences between BERT and MT encoder spaces and how to address them in translation tasks. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Student Research Workshop. redaktör / Jad Kabbara ; Haitao Lin ; Amandalynne Paullada ; Jannis Vamvas. Stroudsburg : The Association for Computational Linguistics, 2021. s. 337-347

@inproceedings{1d9dd4dca38e4631869a4f0901397bac,

title = "On the differences between BERT and MT encoder spaces and how to address them in translation tasks",

abstract = "Various studies show that pretrained language models such as BERT cannot straightforwardly replace encoders in neural machine translation despite their enormous success in other tasks. This is even more astonishing considering the similarities between the architectures. This paper sheds some light on the embedding spaces they create, using average cosine similarity, contextuality metrics and measures for representational similarity for comparison, revealing that BERT and NMT encoder representations look significantly different from one another. In order to address this issue, we propose a supervised transformation from one into the other using explicit alignment and fine-tuning. Our results demonstrate the need for such a transformation to improve the applicability of BERT in MT.",

keywords = "113 Computer and information sciences, 6121 Languages",

author = "Raul Vazquez and Hande Celikkanat and Mathias Creutz and J{\"o}rg Tiedemann",

year = "2021",

month = aug,

doi = "10.18653/v1/2021.acl-srw.35",

language = "English",

isbn = "978-1-954085-55-8",

pages = "337--347",

editor = "Jad Kabbara and Lin, {Haitao } and Paullada, {Amandalynne } and Vamvas, {Jannis }",

booktitle = "Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing",

publisher = "The Association for Computational Linguistics",

address = "United States",

note = "Annual Meeting of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021 ; Conference date: 05-08-2021 Through 06-08-2021",

}

Vazquez , R , Celikkanat, H , Creutz, M & Tiedemann, J 2021, On the differences between BERT and MT encoder spaces and how to address them in translation tasks. i J Kabbara, H Lin, A Paullada & J Vamvas (red), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Student Research Workshop. The Association for Computational Linguistics, Stroudsburg, s. 337-347, Annual Meeting of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing, 05/08/2021. https://doi.org/10.18653/v1/2021.acl-srw.35

On the differences between BERT and MT encoder spaces and how to address them in translation tasks. / Vazquez , Raul ; Celikkanat, Hande ; Creutz, Mathias et al.
Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Student Research Workshop. red. / Jad Kabbara; Haitao Lin; Amandalynne Paullada; Jannis Vamvas. Stroudsburg: The Association for Computational Linguistics, 2021. s. 337-347.

Forskningsoutput: Kapitel i bok/rapport/konferenshandling › Konferensbidrag › Vetenskaplig › Peer review

TY - GEN

T1 - On the differences between BERT and MT encoder spaces and how to address them in translation tasks

AU - Vazquez , Raul

AU - Celikkanat, Hande

AU - Creutz, Mathias

AU - Tiedemann, Jörg

N1 - Conference code: 59/11

PY - 2021/8

Y1 - 2021/8

N2 - Various studies show that pretrained language models such as BERT cannot straightforwardly replace encoders in neural machine translation despite their enormous success in other tasks. This is even more astonishing considering the similarities between the architectures. This paper sheds some light on the embedding spaces they create, using average cosine similarity, contextuality metrics and measures for representational similarity for comparison, revealing that BERT and NMT encoder representations look significantly different from one another. In order to address this issue, we propose a supervised transformation from one into the other using explicit alignment and fine-tuning. Our results demonstrate the need for such a transformation to improve the applicability of BERT in MT.

AB - Various studies show that pretrained language models such as BERT cannot straightforwardly replace encoders in neural machine translation despite their enormous success in other tasks. This is even more astonishing considering the similarities between the architectures. This paper sheds some light on the embedding spaces they create, using average cosine similarity, contextuality metrics and measures for representational similarity for comparison, revealing that BERT and NMT encoder representations look significantly different from one another. In order to address this issue, we propose a supervised transformation from one into the other using explicit alignment and fine-tuning. Our results demonstrate the need for such a transformation to improve the applicability of BERT in MT.

KW - 113 Computer and information sciences

KW - 6121 Languages

U2 - 10.18653/v1/2021.acl-srw.35

DO - 10.18653/v1/2021.acl-srw.35

M3 - Conference contribution

SN - 978-1-954085-55-8

SP - 337

EP - 347

BT - Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing

A2 - Kabbara, Jad

A2 - Lin, Haitao

A2 - Paullada, Amandalynne

A2 - Vamvas, Jannis

PB - The Association for Computational Linguistics

CY - Stroudsburg

T2 - Annual Meeting of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing

Y2 - 5 August 2021 through 6 August 2021

ER -

Vazquez R , Celikkanat H , Creutz M , Tiedemann J. On the differences between BERT and MT encoder spaces and how to address them in translation tasks. I Kabbara J, Lin H, Paullada A, Vamvas J, redaktörer, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Student Research Workshop. Stroudsburg: The Association for Computational Linguistics. 2021. s. 337-347 doi: 10.18653/v1/2021.acl-srw.35

On the differences between BERT and MT encoder spaces and how to address them in translation tasks

Sammanfattning

Vetenskapsgrenar

Åtkomst av dokument

Projekt

FoTran: Found in Translation - Natural Language Understanding with Cross-Lingual Grounding

Citera det här