Constraint Grammar is a hand-crafted Transformer

Anssi Yli-Jyrä

Constraint Grammar is a hand-crafted Transformer

Språkteknologi

Forskningsoutput: Kapitel i bok/rapport/konferenshandling › Konferensbidrag › Vetenskaplig › Peer review

Sammanfattning

Deep neural networks (DNN) and linguistic rules are currently the opposite ends in the scale for NLP technologies. Until recently, it has not been known how to combine these technologies most effectively. Therefore, the technologies have been the object of almost disjoint research communities. In this presentation, I first recall that both Constraint Grammar (CG) and vanilla RNNs have finite-state properties. Then I relate CG to Google’s Transformer architecture (with two kinds of attention) and argue that there are significant similarities between these two seemingly unrelated architectures.

Originalspråk	engelska
Titel på värdpublikation	Proceedings of the NoDaLiDa 2019 Workshop on Constraint Grammar - Methods, Tools and Applications, 30 September 2019, Turku, Finland
Redaktörer	Eckhard Bick, Trond Trosterud
Antal sidor	5
Utgivningsort	Linköping
Förlag	Linköping University Electronic Press
Utgivningsdatum	3 dec. 2019
Sidor	45-49
Artikelnummer	9
ISBN (elektroniskt)	978-91-7929-918-7
Status	Publicerad - 3 dec. 2019
MoE-publikationstyp	A4 Artikel i en konferenspublikation
Evenemang	NoDaLiDa 2019 workshop on Constraint Grammar - Methods, Tools, and Applications - University of Turku, Turku, Finland Varaktighet: 30 sep. 2019 → 30 sep. 2019 https://visl.sdu.dk/nodalida2019.html

Publikationsserier

Namn	NEALT Proceedings Series
Förlag	Linköping University Electronic Press, Linköpings universitet
Nummer	33

Namn	Linköping Electronic Conference Proceedings
Förlag	Linköping University Electronic Press, Linköpings universitet
Nummer	168
ISSN (tryckt)	1650-3686
ISSN (elektroniskt)	1650-3740

Vetenskapsgrenar

113 Data- och informationsvetenskap

Åtkomst av dokument

ecp19168009Slutlig publicerad version, 826 KBLicens: CC BY

http://www.ep.liu.se/ecp/168/009/ecp19168009.pdfLicens: CC BY

Andra filer och länkar

PDF

1 Slutfört

ADEQSYNTAX: Käytettävä äärellistilainen malli adekvaatille kieliopilliselle kompleksisuudelle
Yli-Jyrä, A.
Suomen tietokirjailijat
01/09/2013 → 30/04/2019
Projekt: Forskningsprojekt

Citera det här

Yli-Jyrä, A. (2019). Constraint Grammar is a hand-crafted Transformer. I E. Bick, & T. Trosterud (Red.), Proceedings of the NoDaLiDa 2019 Workshop on Constraint Grammar - Methods, Tools and Applications, 30 September 2019, Turku, Finland (s. 45-49). [9] (NEALT Proceedings Series; Nr. 33), (Linköping Electronic Conference Proceedings; Nr. 168). Linköping University Electronic Press. http://www.ep.liu.se/ecp/168/009/ecp19168009.pdf

Yli-Jyrä, Anssi. / Constraint Grammar is a hand-crafted Transformer. Proceedings of the NoDaLiDa 2019 Workshop on Constraint Grammar - Methods, Tools and Applications, 30 September 2019, Turku, Finland. redaktör / Eckhard Bick ; Trond Trosterud. Linköping : Linköping University Electronic Press, 2019. s. 45-49 (NEALT Proceedings Series; 33). (Linköping Electronic Conference Proceedings; 168).

@inproceedings{3762585d0b1a479cb8e8a695f6e2bf3d,

title = "Constraint Grammar is a hand-crafted Transformer",

abstract = "Deep neural networks (DNN) and linguistic rules are currently the opposite ends in the scale for NLP technologies. Until recently, it has not been known how to combine these technologies most effectively. Therefore, the technologies have been the object of almost disjoint research communities. In this presentation, I first recall that both Constraint Grammar (CG) and vanilla RNNs have finite-state properties. Then I relate CG to Google{\textquoteright}s Transformer architecture (with two kinds of attention) and argue that there are significant similarities between these two seemingly unrelated architectures.",

keywords = "113 Computer and information sciences, constraint grammar, finite-state capacity, recurrent neural networks, self-attention, attention, rule conditions, Transformer",

author = "Anssi Yli-Jyr{\"a}",

year = "2019",

month = dec,

day = "3",

language = "English",

series = "NEALT Proceedings Series",

publisher = "Link{\"o}ping University Electronic Press",

number = "33",

pages = "45--49",

editor = "Eckhard Bick and Trond Trosterud",

booktitle = "Proceedings of the NoDaLiDa 2019 Workshop on Constraint Grammar - Methods, Tools and Applications, 30 September 2019, Turku, Finland",

address = "Sweden",

note = "null ; Conference date: 30-09-2019 Through 30-09-2019",

url = "https://visl.sdu.dk/nodalida2019.html",

}

Yli-Jyrä, A 2019, Constraint Grammar is a hand-crafted Transformer. i E Bick & T Trosterud (red), Proceedings of the NoDaLiDa 2019 Workshop on Constraint Grammar - Methods, Tools and Applications, 30 September 2019, Turku, Finland., 9, NEALT Proceedings Series, nr. 33, Linköping Electronic Conference Proceedings, nr. 168, Linköping University Electronic Press, Linköping, s. 45-49, NoDaLiDa 2019 workshop on Constraint Grammar - Methods, Tools, and Applications, Turku, Finland, 30/09/2019. <http://www.ep.liu.se/ecp/168/009/ecp19168009.pdf>

Constraint Grammar is a hand-crafted Transformer. / Yli-Jyrä, Anssi.

Proceedings of the NoDaLiDa 2019 Workshop on Constraint Grammar - Methods, Tools and Applications, 30 September 2019, Turku, Finland. red. / Eckhard Bick; Trond Trosterud. Linköping : Linköping University Electronic Press, 2019. s. 45-49 9 (NEALT Proceedings Series; Nr. 33), (Linköping Electronic Conference Proceedings; Nr. 168).

Forskningsoutput: Kapitel i bok/rapport/konferenshandling › Konferensbidrag › Vetenskaplig › Peer review

TY - GEN

T1 - Constraint Grammar is a hand-crafted Transformer

AU - Yli-Jyrä, Anssi

PY - 2019/12/3

Y1 - 2019/12/3

N2 - Deep neural networks (DNN) and linguistic rules are currently the opposite ends in the scale for NLP technologies. Until recently, it has not been known how to combine these technologies most effectively. Therefore, the technologies have been the object of almost disjoint research communities. In this presentation, I first recall that both Constraint Grammar (CG) and vanilla RNNs have finite-state properties. Then I relate CG to Google’s Transformer architecture (with two kinds of attention) and argue that there are significant similarities between these two seemingly unrelated architectures.

AB - Deep neural networks (DNN) and linguistic rules are currently the opposite ends in the scale for NLP technologies. Until recently, it has not been known how to combine these technologies most effectively. Therefore, the technologies have been the object of almost disjoint research communities. In this presentation, I first recall that both Constraint Grammar (CG) and vanilla RNNs have finite-state properties. Then I relate CG to Google’s Transformer architecture (with two kinds of attention) and argue that there are significant similarities between these two seemingly unrelated architectures.

KW - 113 Computer and information sciences

KW - constraint grammar

KW - finite-state capacity

KW - recurrent neural networks

KW - self-attention

KW - attention

KW - rule conditions

KW - Transformer

UR - http://www.ep.liu.se/ecp/article.asp?issue=168&article=009&volume=

M3 - Conference contribution

T3 - NEALT Proceedings Series

SP - 45

EP - 49

BT - Proceedings of the NoDaLiDa 2019 Workshop on Constraint Grammar - Methods, Tools and Applications, 30 September 2019, Turku, Finland

A2 - Bick, Eckhard

A2 - Trosterud, Trond

PB - Linköping University Electronic Press

CY - Linköping

Y2 - 30 September 2019 through 30 September 2019

ER -

Yli-Jyrä A. Constraint Grammar is a hand-crafted Transformer. I Bick E, Trosterud T, redaktörer, Proceedings of the NoDaLiDa 2019 Workshop on Constraint Grammar - Methods, Tools and Applications, 30 September 2019, Turku, Finland. Linköping: Linköping University Electronic Press. 2019. s. 45-49. 9. (NEALT Proceedings Series; 33). (Linköping Electronic Conference Proceedings; 168).

Constraint Grammar is a hand-crafted Transformer

Sammanfattning

Publikationsserier

Vetenskapsgrenar

Åtkomst av dokument

Andra filer och länkar

Projekt

ADEQSYNTAX: Käytettävä äärellistilainen malli adekvaatille kieliopilliselle kompleksisuudelle

Citera det här