Constraint Grammar is a hand-crafted Transformer

Anssi Yli-Jyrä

Constraint Grammar is a hand-crafted Transformer

Anssi Yli-Jyrä

Language Technology

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Scientific › peer-review

Abstract

Deep neural networks (DNN) and linguistic rules are currently the opposite ends in the scale for NLP technologies. Until recently, it has not been known how to combine these technologies most effectively. Therefore, the technologies have been the object of almost disjoint research communities. In this presentation, I first recall that both Constraint Grammar (CG) and vanilla RNNs have finite-state properties. Then I relate CG to Google’s Transformer architecture (with two kinds of attention) and argue that there are significant similarities between these two seemingly unrelated architectures.

Original language	English
Title of host publication	Proceedings of the NoDaLiDa 2019 Workshop on Constraint Grammar - Methods, Tools and Applications, 30 September 2019, Turku, Finland
Editors	Eckhard Bick, Trond Trosterud
Number of pages	5
Place of Publication	Linköping
Publisher	Linköping University Electronic Press
Publication date	3 Dec 2019
Pages	45-49
Article number	9
ISBN (Electronic)	978-91-7929-918-7
Publication status	Published - 3 Dec 2019
MoE publication type	A4 Article in conference proceedings
Event	NoDaLiDa 2019 workshop on Constraint Grammar - Methods, Tools, and Applications - University of Turku, Turku, Finland Duration: 30 Sept 2019 → 30 Sept 2019 https://visl.sdu.dk/nodalida2019.html

Publication series

Name	NEALT Proceedings Series
Publisher	Linköping University Electronic Press, Linköpings universitet
Number	33

Name	Linköping Electronic Conference Proceedings
Publisher	Linköping University Electronic Press, Linköpings universitet
Number	168
ISSN (Print)	1650-3686
ISSN (Electronic)	1650-3740

Fields of Science

113 Computer and information sciences
constraint grammar
finite-state capacity
recurrent neural networks
self-attention
attention
rule conditions
Transformer

Access to Document

ecp19168009Final published version, 826 KBLicence: CC BY

http://www.ep.liu.se/ecp/168/009/ecp19168009.pdfLicence: CC BY

Cite this

Yli-Jyrä, A. (2019). Constraint Grammar is a hand-crafted Transformer. In E. Bick, & T. Trosterud (Eds.), Proceedings of the NoDaLiDa 2019 Workshop on Constraint Grammar - Methods, Tools and Applications, 30 September 2019, Turku, Finland (pp. 45-49). Article 9 (NEALT Proceedings Series; No. 33), (Linköping Electronic Conference Proceedings; No. 168). Linköping University Electronic Press. http://www.ep.liu.se/ecp/168/009/ecp19168009.pdf

Yli-Jyrä, Anssi. / Constraint Grammar is a hand-crafted Transformer. Proceedings of the NoDaLiDa 2019 Workshop on Constraint Grammar - Methods, Tools and Applications, 30 September 2019, Turku, Finland. editor / Eckhard Bick ; Trond Trosterud. Linköping : Linköping University Electronic Press, 2019. pp. 45-49 (NEALT Proceedings Series; 33). (Linköping Electronic Conference Proceedings; 168).

@inproceedings{3762585d0b1a479cb8e8a695f6e2bf3d,

title = "Constraint Grammar is a hand-crafted Transformer",

abstract = "Deep neural networks (DNN) and linguistic rules are currently the opposite ends in the scale for NLP technologies. Until recently, it has not been known how to combine these technologies most effectively. Therefore, the technologies have been the object of almost disjoint research communities. In this presentation, I first recall that both Constraint Grammar (CG) and vanilla RNNs have finite-state properties. Then I relate CG to Google{\textquoteright}s Transformer architecture (with two kinds of attention) and argue that there are significant similarities between these two seemingly unrelated architectures.",

keywords = "113 Computer and information sciences, constraint grammar, finite-state capacity, recurrent neural networks, self-attention, attention, rule conditions, Transformer",

author = "Anssi Yli-Jyr{\"a}",

year = "2019",

month = dec,

day = "3",

language = "English",

series = "NEALT Proceedings Series",

publisher = "Link{\"o}ping University Electronic Press",

number = "33",

pages = "45--49",

editor = "Eckhard Bick and Trond Trosterud",

booktitle = "Proceedings of the NoDaLiDa 2019 Workshop on Constraint Grammar - Methods, Tools and Applications, 30 September 2019, Turku, Finland",

address = "Sweden",

note = "NoDaLiDa 2019 workshop on Constraint Grammar - Methods, Tools, and Applications ; Conference date: 30-09-2019 Through 30-09-2019",

url = "https://visl.sdu.dk/nodalida2019.html",

}

Yli-Jyrä, A 2019, Constraint Grammar is a hand-crafted Transformer. in E Bick & T Trosterud (eds), Proceedings of the NoDaLiDa 2019 Workshop on Constraint Grammar - Methods, Tools and Applications, 30 September 2019, Turku, Finland., 9, NEALT Proceedings Series, no. 33, Linköping Electronic Conference Proceedings, no. 168, Linköping University Electronic Press, Linköping, pp. 45-49, NoDaLiDa 2019 workshop on Constraint Grammar - Methods, Tools, and Applications, Turku, Finland, 30/09/2019. <http://www.ep.liu.se/ecp/168/009/ecp19168009.pdf>

Constraint Grammar is a hand-crafted Transformer. / Yli-Jyrä, Anssi.
Proceedings of the NoDaLiDa 2019 Workshop on Constraint Grammar - Methods, Tools and Applications, 30 September 2019, Turku, Finland. ed. / Eckhard Bick; Trond Trosterud. Linköping: Linköping University Electronic Press, 2019. p. 45-49 9 (NEALT Proceedings Series; No. 33), (Linköping Electronic Conference Proceedings; No. 168).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Scientific › peer-review

TY - GEN

T1 - Constraint Grammar is a hand-crafted Transformer

AU - Yli-Jyrä, Anssi

PY - 2019/12/3

Y1 - 2019/12/3

N2 - Deep neural networks (DNN) and linguistic rules are currently the opposite ends in the scale for NLP technologies. Until recently, it has not been known how to combine these technologies most effectively. Therefore, the technologies have been the object of almost disjoint research communities. In this presentation, I first recall that both Constraint Grammar (CG) and vanilla RNNs have finite-state properties. Then I relate CG to Google’s Transformer architecture (with two kinds of attention) and argue that there are significant similarities between these two seemingly unrelated architectures.

AB - Deep neural networks (DNN) and linguistic rules are currently the opposite ends in the scale for NLP technologies. Until recently, it has not been known how to combine these technologies most effectively. Therefore, the technologies have been the object of almost disjoint research communities. In this presentation, I first recall that both Constraint Grammar (CG) and vanilla RNNs have finite-state properties. Then I relate CG to Google’s Transformer architecture (with two kinds of attention) and argue that there are significant similarities between these two seemingly unrelated architectures.

KW - 113 Computer and information sciences

KW - constraint grammar

KW - finite-state capacity

KW - recurrent neural networks

KW - self-attention

KW - attention

KW - rule conditions

KW - Transformer

UR - http://www.ep.liu.se/ecp/article.asp?issue=168&article=009&volume=

M3 - Conference contribution

T3 - NEALT Proceedings Series

SP - 45

EP - 49

BT - Proceedings of the NoDaLiDa 2019 Workshop on Constraint Grammar - Methods, Tools and Applications, 30 September 2019, Turku, Finland

A2 - Bick, Eckhard

A2 - Trosterud, Trond

PB - Linköping University Electronic Press

CY - Linköping

T2 - NoDaLiDa 2019 workshop on Constraint Grammar - Methods, Tools, and Applications

Y2 - 30 September 2019 through 30 September 2019

ER -

Yli-Jyrä A. Constraint Grammar is a hand-crafted Transformer. In Bick E, Trosterud T, editors, Proceedings of the NoDaLiDa 2019 Workshop on Constraint Grammar - Methods, Tools and Applications, 30 September 2019, Turku, Finland. Linköping: Linköping University Electronic Press. 2019. p. 45-49. 9. (NEALT Proceedings Series; 33). (Linköping Electronic Conference Proceedings; 168).

Constraint Grammar is a hand-crafted Transformer

Abstract

Publication series

Fields of Science

Access to Document

Other files and links

ADEQSYNTAX: A Usable Finite-State Model for Adequate Syntactic Complexity

Cite this

Constraint Grammar is a hand-crafted Transformer

Abstract

Publication series

Fields of Science

Access to Document

Other files and links

Projects

ADEQSYNTAX: A Usable Finite-State Model for Adequate Syntactic Complexity

Cite this