On the differences between BERT and MT encoder spaces and how to address them in translation tasks

Raul Vazquez; Hande Celikkanat; Mathias Creutz; Jörg Tiedemann

doi:10.18653/v1/2021.acl-srw.35

On the differences between BERT and MT encoder spaces and how to address them in translation tasks

Raul Vazquez , Hande Celikkanat, Mathias Creutz, Jörg Tiedemann

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Scientific › peer-review

Abstract

Various studies show that pretrained language models such as BERT cannot straightforwardly replace encoders in neural machine translation despite their enormous success in other tasks. This is even more astonishing considering the similarities between the architectures. This paper sheds some light on the embedding spaces they create, using average cosine similarity, contextuality metrics and measures for representational similarity for comparison, revealing that BERT and NMT encoder representations look significantly different from one another. In order to address this issue, we propose a supervised transformation from one into the other using explicit alignment and fine-tuning. Our results demonstrate the need for such a transformation to improve the applicability of BERT in MT.

Original language	English
Title of host publication	Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing : Student Research Workshop
Editors	Jad Kabbara, Haitao Lin, Amandalynne Paullada, Jannis Vamvas
Number of pages	11
Place of Publication	Stroudsburg
Publisher	The Association for Computational Linguistics
Publication date	Aug 2021
Pages	337-347
ISBN (Print)	978-1-954085-55-8
DOIs	https://doi.org/10.18653/v1/2021.acl-srw.35
Publication status	Published - Aug 2021
MoE publication type	A4 Article in conference proceedings
Event	Annual Meeting of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing - Bangkok [Online event] Duration: 5 Aug 2021 → 6 Aug 2021 Conference number: 59/11

Fields of Science

113 Computer and information sciences
6121 Languages

Access to Document

10.18653/v1/2021.acl-srw.35Licence: CC BY

2021.acl-srw.35Final published version, 0.99 MBLicence: CC BY

FoTran: Found in Translation - Natural Language Understanding with Cross-Lingual Grounding
Tiedemann, J. (Project manager), Attieh, J. (Participant), Aulamo, M. (Participant), Celikkanat, H. (Participant), De Gibert Bonet, O. (Participant), Grönroos, S.-A. (Participant), Mickus, T. (Participant), Scherrer, Y. (Participant), Silfverberg, M. (Participant), Talman, A. (Participant), Vazquez , R. (Participant), Virpioja, S. P. (Participant), Yli-Jyrä, A. (Participant), Celikkanat, H. (Participant), Raganato, A. (Participant), Silfverberg, M. (Participant), Sulubacak, U. (Participant) & Vazquez , R. (Participant)
01/09/2018 → 31/03/2024
Project: The European Research Council: Consolidator Grant (H2020-ERC-COG)

Cite this

Vazquez , R., Celikkanat, H., Creutz, M., & Tiedemann, J. (2021). On the differences between BERT and MT encoder spaces and how to address them in translation tasks. In J. Kabbara, H. Lin, A. Paullada, & J. Vamvas (Eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Student Research Workshop (pp. 337-347). The Association for Computational Linguistics. https://doi.org/10.18653/v1/2021.acl-srw.35

Vazquez , Raul ; Celikkanat, Hande ; Creutz, Mathias et al. / On the differences between BERT and MT encoder spaces and how to address them in translation tasks. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Student Research Workshop. editor / Jad Kabbara ; Haitao Lin ; Amandalynne Paullada ; Jannis Vamvas. Stroudsburg : The Association for Computational Linguistics, 2021. pp. 337-347

@inproceedings{1d9dd4dca38e4631869a4f0901397bac,

title = "On the differences between BERT and MT encoder spaces and how to address them in translation tasks",

abstract = "Various studies show that pretrained language models such as BERT cannot straightforwardly replace encoders in neural machine translation despite their enormous success in other tasks. This is even more astonishing considering the similarities between the architectures. This paper sheds some light on the embedding spaces they create, using average cosine similarity, contextuality metrics and measures for representational similarity for comparison, revealing that BERT and NMT encoder representations look significantly different from one another. In order to address this issue, we propose a supervised transformation from one into the other using explicit alignment and fine-tuning. Our results demonstrate the need for such a transformation to improve the applicability of BERT in MT.",

keywords = "113 Computer and information sciences, 6121 Languages",

author = "Raul Vazquez and Hande Celikkanat and Mathias Creutz and J{\"o}rg Tiedemann",

year = "2021",

month = aug,

doi = "10.18653/v1/2021.acl-srw.35",

language = "English",

isbn = "978-1-954085-55-8",

pages = "337--347",

editor = "Jad Kabbara and Lin, \{Haitao \} and Paullada, \{Amandalynne \} and Vamvas, \{Jannis \}",

booktitle = "Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing",

publisher = "The Association for Computational Linguistics",

address = "United States",

note = "Annual Meeting of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021 ; Conference date: 05-08-2021 Through 06-08-2021",

}

Vazquez , R , Celikkanat, H , Creutz, M & Tiedemann, J 2021, On the differences between BERT and MT encoder spaces and how to address them in translation tasks. in J Kabbara, H Lin, A Paullada & J Vamvas (eds), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Student Research Workshop. The Association for Computational Linguistics, Stroudsburg, pp. 337-347, Annual Meeting of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing, 05/08/2021. https://doi.org/10.18653/v1/2021.acl-srw.35

On the differences between BERT and MT encoder spaces and how to address them in translation tasks. / Vazquez , Raul ; Celikkanat, Hande ; Creutz, Mathias et al.
Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Student Research Workshop. ed. / Jad Kabbara; Haitao Lin; Amandalynne Paullada; Jannis Vamvas. Stroudsburg: The Association for Computational Linguistics, 2021. p. 337-347.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Scientific › peer-review

TY - GEN

T1 - On the differences between BERT and MT encoder spaces and how to address them in translation tasks

AU - Vazquez , Raul

AU - Celikkanat, Hande

AU - Creutz, Mathias

AU - Tiedemann, Jörg

N1 - Conference code: 59/11

PY - 2021/8

Y1 - 2021/8

N2 - Various studies show that pretrained language models such as BERT cannot straightforwardly replace encoders in neural machine translation despite their enormous success in other tasks. This is even more astonishing considering the similarities between the architectures. This paper sheds some light on the embedding spaces they create, using average cosine similarity, contextuality metrics and measures for representational similarity for comparison, revealing that BERT and NMT encoder representations look significantly different from one another. In order to address this issue, we propose a supervised transformation from one into the other using explicit alignment and fine-tuning. Our results demonstrate the need for such a transformation to improve the applicability of BERT in MT.

AB - Various studies show that pretrained language models such as BERT cannot straightforwardly replace encoders in neural machine translation despite their enormous success in other tasks. This is even more astonishing considering the similarities between the architectures. This paper sheds some light on the embedding spaces they create, using average cosine similarity, contextuality metrics and measures for representational similarity for comparison, revealing that BERT and NMT encoder representations look significantly different from one another. In order to address this issue, we propose a supervised transformation from one into the other using explicit alignment and fine-tuning. Our results demonstrate the need for such a transformation to improve the applicability of BERT in MT.

KW - 113 Computer and information sciences

KW - 6121 Languages

U2 - 10.18653/v1/2021.acl-srw.35

DO - 10.18653/v1/2021.acl-srw.35

M3 - Conference contribution

SN - 978-1-954085-55-8

SP - 337

EP - 347

BT - Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing

A2 - Kabbara, Jad

A2 - Lin, Haitao

A2 - Paullada, Amandalynne

A2 - Vamvas, Jannis

PB - The Association for Computational Linguistics

CY - Stroudsburg

T2 - Annual Meeting of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing

Y2 - 5 August 2021 through 6 August 2021

ER -

Vazquez R , Celikkanat H , Creutz M , Tiedemann J. On the differences between BERT and MT encoder spaces and how to address them in translation tasks. In Kabbara J, Lin H, Paullada A, Vamvas J, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Student Research Workshop. Stroudsburg: The Association for Computational Linguistics. 2021. p. 337-347 doi: 10.18653/v1/2021.acl-srw.35

On the differences between BERT and MT encoder spaces and how to address them in translation tasks

Abstract

Fields of Science

Access to Document

Projects

FoTran: Found in Translation - Natural Language Understanding with Cross-Lingual Grounding

Cite this