Tracking the Traces of Passivization and Negation in Contextualized Representations

Hande Celikkanat; Sami Virpioja; Jörg Tiedemann; Marianna Apidianaki

doi:10.18653/v1/2020.blackboxnlp-1.13

Tracking the Traces of Passivization and Negation in Contextualized Representations

Hande Celikkanat, Sami Virpioja, Jörg Tiedemann, Marianna Apidianaki

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Scientific › peer-review

Abstract

Contextualized word representations encode rich information about syntax and semantics, alongside specificities of each context of use. While contextual variation does not always reflect actual meaning shifts, it can still reduce the similarity of embeddings for word instances having the same meaning. We explore the imprint of two specific linguistic alternations, namely passivization and negation, on the representations generated by neural models trained with two different objectives: masked language modeling and translation. Our exploration methodology is inspired by an approach previously proposed for removing societal biases from word vectors. We show that passivization and negation leave their traces on the representations, and that neutralizing this information leads to more similar embeddings for words that should preserve their meaning in the transformation. We also find clear differences in how the respective features generalize across datasets.

Original language	English
Title of host publication	Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP
Editors	Afra Alishahi, Yonatan Belinkov, Grzegorz Chrupała, Dieuwke Hupkes, Yuval Pinter, Hassan Sajjad
Number of pages	13
Place of Publication	Stroudsburg
Publisher	The Association for Computational Linguistics
Publication date	20 Nov 2020
Pages	136-148
ISBN (Electronic)	978-1-952148-86-6
DOIs	https://doi.org/10.18653/v1/2020.blackboxnlp-1.13
Publication status	Published - 20 Nov 2020
MoE publication type	A4 Article in conference proceedings
Event	BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP - Online event Duration: 20 Nov 2020 → 20 Nov 2020 Conference number: 3

Fields of Science

6121 Languages
113 Computer and information sciences

Access to Document

10.18653/v1/2020.blackboxnlp-1.13Licence: CC BY

2020.blackboxnlp-1.13Final published version, 5.33 MBLicence: CC BY

FoTran: Found in Translation - Natural Language Understanding with Cross-Lingual Grounding
Tiedemann, J. (Project manager), Attieh, J. (Participant), Aulamo, M. (Participant), Celikkanat, H. (Participant), De Gibert Bonet, O. (Participant), Grönroos, S.-A. (Participant), Mickus, T. (Participant), Scherrer, Y. (Participant), Silfverberg, M. (Participant), Talman, A. (Participant), Vazquez , R. (Participant), Virpioja, S. P. (Participant), Yli-Jyrä, A. (Participant), Celikkanat, H. (Participant), Raganato, A. (Participant), Silfverberg, M. (Participant), Sulubacak, U. (Participant) & Vazquez , R. (Participant)
01/09/2018 → 31/03/2024
Project: The European Research Council: Consolidator Grant (H2020-ERC-COG)

Cite this

Celikkanat, H., Virpioja, S., Tiedemann, J., & Apidianaki, M. (2020). Tracking the Traces of Passivization and Negation in Contextualized Representations. In A. Alishahi, Y. Belinkov, G. Chrupała, D. Hupkes, Y. Pinter, & H. Sajjad (Eds.), Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (pp. 136-148). The Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.blackboxnlp-1.13

Celikkanat, Hande ; Virpioja, Sami ; Tiedemann, Jörg et al. / Tracking the Traces of Passivization and Negation in Contextualized Representations. Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP. editor / Afra Alishahi ; Yonatan Belinkov ; Grzegorz Chrupała ; Dieuwke Hupkes ; Yuval Pinter ; Hassan Sajjad. Stroudsburg : The Association for Computational Linguistics, 2020. pp. 136-148

@inproceedings{09d629fd513e4b8bab0f4ccdf39c9672,

title = "Tracking the Traces of Passivization and Negation in Contextualized Representations",

abstract = "Contextualized word representations encode rich information about syntax and semantics, alongside specificities of each context of use. While contextual variation does not always reflect actual meaning shifts, it can still reduce the similarity of embeddings for word instances having the same meaning. We explore the imprint of two specific linguistic alternations, namely passivization and negation, on the representations generated by neural models trained with two different objectives: masked language modeling and translation. Our exploration methodology is inspired by an approach previously proposed for removing societal biases from word vectors. We show that passivization and negation leave their traces on the representations, and that neutralizing this information leads to more similar embeddings for words that should preserve their meaning in the transformation. We also find clear differences in how the respective features generalize across datasets.",

keywords = "6121 Languages, 113 Computer and information sciences",

author = "Hande Celikkanat and Sami Virpioja and J{\"o}rg Tiedemann and Marianna Apidianaki",

year = "2020",

month = nov,

day = "20",

doi = "10.18653/v1/2020.blackboxnlp-1.13",

language = "English",

pages = "136--148",

editor = "Alishahi, \{Afra \} and Belinkov, \{Yonatan \} and Chrupa{\l}a, \{Grzegorz \} and Hupkes, \{Dieuwke \} and Pinter, \{Yuval \} and Sajjad, \{Hassan \}",

booktitle = "Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP",

publisher = "The Association for Computational Linguistics",

address = "United States",

note = "BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP ; Conference date: 20-11-2020 Through 20-11-2020",

}

Celikkanat, H , Virpioja, S , Tiedemann, J & Apidianaki, M 2020, Tracking the Traces of Passivization and Negation in Contextualized Representations. in A Alishahi, Y Belinkov, G Chrupała, D Hupkes, Y Pinter & H Sajjad (eds), Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP. The Association for Computational Linguistics, Stroudsburg, pp. 136-148, BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, 20/11/2020. https://doi.org/10.18653/v1/2020.blackboxnlp-1.13

Tracking the Traces of Passivization and Negation in Contextualized Representations. / Celikkanat, Hande ; Virpioja, Sami ; Tiedemann, Jörg et al.
Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP. ed. / Afra Alishahi; Yonatan Belinkov; Grzegorz Chrupała; Dieuwke Hupkes; Yuval Pinter; Hassan Sajjad. Stroudsburg: The Association for Computational Linguistics, 2020. p. 136-148.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Scientific › peer-review

TY - GEN

T1 - Tracking the Traces of Passivization and Negation in Contextualized Representations

AU - Celikkanat, Hande

AU - Virpioja, Sami

AU - Tiedemann, Jörg

AU - Apidianaki, Marianna

N1 - Conference code: 3

PY - 2020/11/20

Y1 - 2020/11/20

N2 - Contextualized word representations encode rich information about syntax and semantics, alongside specificities of each context of use. While contextual variation does not always reflect actual meaning shifts, it can still reduce the similarity of embeddings for word instances having the same meaning. We explore the imprint of two specific linguistic alternations, namely passivization and negation, on the representations generated by neural models trained with two different objectives: masked language modeling and translation. Our exploration methodology is inspired by an approach previously proposed for removing societal biases from word vectors. We show that passivization and negation leave their traces on the representations, and that neutralizing this information leads to more similar embeddings for words that should preserve their meaning in the transformation. We also find clear differences in how the respective features generalize across datasets.

AB - Contextualized word representations encode rich information about syntax and semantics, alongside specificities of each context of use. While contextual variation does not always reflect actual meaning shifts, it can still reduce the similarity of embeddings for word instances having the same meaning. We explore the imprint of two specific linguistic alternations, namely passivization and negation, on the representations generated by neural models trained with two different objectives: masked language modeling and translation. Our exploration methodology is inspired by an approach previously proposed for removing societal biases from word vectors. We show that passivization and negation leave their traces on the representations, and that neutralizing this information leads to more similar embeddings for words that should preserve their meaning in the transformation. We also find clear differences in how the respective features generalize across datasets.

KW - 6121 Languages

KW - 113 Computer and information sciences

U2 - 10.18653/v1/2020.blackboxnlp-1.13

DO - 10.18653/v1/2020.blackboxnlp-1.13

M3 - Conference contribution

SP - 136

EP - 148

BT - Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP

A2 - Alishahi, Afra

A2 - Belinkov, Yonatan

A2 - Chrupała, Grzegorz

A2 - Hupkes, Dieuwke

A2 - Pinter, Yuval

A2 - Sajjad, Hassan

PB - The Association for Computational Linguistics

CY - Stroudsburg

T2 - BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP

Y2 - 20 November 2020 through 20 November 2020

ER -

Celikkanat H , Virpioja S , Tiedemann J, Apidianaki M. Tracking the Traces of Passivization and Negation in Contextualized Representations. In Alishahi A, Belinkov Y, Chrupała G, Hupkes D, Pinter Y, Sajjad H, editors, Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP. Stroudsburg: The Association for Computational Linguistics. 2020. p. 136-148 doi: 10.18653/v1/2020.blackboxnlp-1.13

Tracking the Traces of Passivization and Negation in Contextualized Representations

Abstract

Fields of Science

Access to Document

Projects

FoTran: Found in Translation - Natural Language Understanding with Cross-Lingual Grounding

Cite this