What does BERT learn about prosody?

Sofoklis Kakouros, Johannah O'Mahony

Tutkimustuotos: Artikkeli kirjassa/raportissa/konferenssijulkaisussaKonferenssiartikkeliTieteellinenvertaisarvioitu

Abstrakti

Language models have become nearly ubiquitous in natural language processing applications achieving state-of-the-art results in many tasks including prosody. As the model design does not define predetermined linguistic targets during training but rather aims at learning generalized representations of the language, analyzing and interpreting the representations that models implicitly capture is important in bridging the gap between interpretability and model performance. Several studies have explored the linguistic information that models capture providing some insights on their representational capacity. However, the current studies have not explored whether prosody is part of the structural information of the language that models learn. In this work, we perform a series of experiments on BERT probing the representations captured at different layers. Our results show that information about prosodic prominence spans across many layers but is mostly focused in middle layers suggesting that BERT relies mostly on syntactic and semantic information.
Alkuperäiskielienglanti
OtsikkoProceedings of the 20th International Congress of Phonetic Sciences (ICPhS 2023)
ToimittajatRadek Skarnitzl, Jan Volín
Sivumäärä5
JulkaisupaikkaPrague
KustantajaGUARANT International spol. s r.o.
Julkaisupäiväelok. 2023
Sivut1454-1458
ISBN (elektroninen)9788090811423
TilaJulkaistu - elok. 2023
OKM-julkaisutyyppiA4 Artikkeli konferenssijulkaisuussa
TapahtumaInternational Congress of Phonetic Sciences - Prague Congress Center, Prague, Tšekki
Kesto: 7 elok. 202311 elok. 2023
Konferenssinumero: 20
https://www.icphs2023.org/

Tieteenalat

  • 213 Sähkö-, automaatio- ja tietoliikennetekniikka, elektroniikka
  • 113 Tietojenkäsittely- ja informaatiotieteet
  • 6121 Kielitieteet

Siteeraa tätä