Finnish ASR with Deep Transformer Models

Abhilash Jain; Aku Rouhe; Stig-Arne Grönroos; Mikko Kurimo

doi:10.21437/Interspeech.2020-1784

Finnish ASR with Deep Transformer Models

Abhilash Jain, Aku Rouhe, Stig-Arne Grönroos, Mikko Kurimo

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Scientific › peer-review

Abstract

Recently, BERT and Transformer-XL based architectures have achieved strong results in a range of NLP applications. In this paper, we explore Transformer architectures—BERT and
Transformer-XL—as a language model for a Finnish ASR task with different rescoring schemes. We achieve strong results in both an intrinsic and an extrinsic task with Transformer-XL. Achieving 29% better perplexity and 3% better WER than our previous best LSTM-based approach. We also introduce a novel three-pass decoding scheme
which improves the ASR performance by 8%. To the best of our knowledge, this is also the first work (i) to formulate an alpha smoothing framework to use the non-autoregressive BERT language model for an ASR task, and (ii) to explore sub-word units with Transformer-XL for an agglutinative language like Finnish.

Original language	English
Title of host publication	Proceedings of Interspeech 2020
Number of pages	5
Place of Publication	Baixas
Publisher	ISCA - International Speech Communication Association
Publication date	2020
Pages	3630-3634
DOIs	https://doi.org/10.21437/Interspeech.2020-1784
Publication status	Published - 2020
Externally published	Yes
MoE publication type	A4 Article in conference proceedings
Event	Interspeech 2020 - [Virtual conference] Duration: 25 Oct 2020 → 29 Oct 2020 http://www.interspeech2020.org

Publication series

Name	Interspeech
Publisher	ISCA
ISSN (Electronic)	2308-457X

Fields of Science

6121 Languages
113 Computer and information sciences

Access to Document

10.21437/Interspeech.2020-1784Licence: Unspecified

jain20b_interspeechFinal published version, 278 KBLicence: Unspecified

Cite this

@inproceedings{8f276b1f29e54422bcab3bae0c14a258,

title = "Finnish ASR with Deep Transformer Models",

abstract = "Recently, BERT and Transformer-XL based architectures have achieved strong results in a range of NLP applications. In this paper, we explore Transformer architectures—BERT andTransformer-XL—as a language model for a Finnish ASR task with different rescoring schemes. We achieve strong results in both an intrinsic and an extrinsic task with Transformer-XL. Achieving 29% better perplexity and 3% better WER than our previous best LSTM-based approach. We also introduce a novel three-pass decoding schemewhich improves the ASR performance by 8%. To the best of our knowledge, this is also the first work (i) to formulate an alpha smoothing framework to use the non-autoregressive BERT language model for an ASR task, and (ii) to explore sub-word units with Transformer-XL for an agglutinative language like Finnish.",

keywords = "6121 Languages, 113 Computer and information sciences",

author = "Abhilash Jain and Aku Rouhe and Stig-Arne Gr{\"o}nroos and Mikko Kurimo",

year = "2020",

doi = "10.21437/Interspeech.2020-1784",

language = "English",

series = "Interspeech",

publisher = "ISCA - International Speech Communication Association ",

pages = "3630--3634",

booktitle = "Proceedings of Interspeech 2020",

address = "Unknown",

note = "Interspeech 2020 ; Conference date: 25-10-2020 Through 29-10-2020",

url = "http://www.interspeech2020.org",

}

TY - GEN

T1 - Finnish ASR with Deep Transformer Models

AU - Jain, Abhilash

AU - Rouhe, Aku

AU - Grönroos, Stig-Arne

AU - Kurimo, Mikko

PY - 2020

Y1 - 2020

N2 - Recently, BERT and Transformer-XL based architectures have achieved strong results in a range of NLP applications. In this paper, we explore Transformer architectures—BERT andTransformer-XL—as a language model for a Finnish ASR task with different rescoring schemes. We achieve strong results in both an intrinsic and an extrinsic task with Transformer-XL. Achieving 29% better perplexity and 3% better WER than our previous best LSTM-based approach. We also introduce a novel three-pass decoding schemewhich improves the ASR performance by 8%. To the best of our knowledge, this is also the first work (i) to formulate an alpha smoothing framework to use the non-autoregressive BERT language model for an ASR task, and (ii) to explore sub-word units with Transformer-XL for an agglutinative language like Finnish.

AB - Recently, BERT and Transformer-XL based architectures have achieved strong results in a range of NLP applications. In this paper, we explore Transformer architectures—BERT andTransformer-XL—as a language model for a Finnish ASR task with different rescoring schemes. We achieve strong results in both an intrinsic and an extrinsic task with Transformer-XL. Achieving 29% better perplexity and 3% better WER than our previous best LSTM-based approach. We also introduce a novel three-pass decoding schemewhich improves the ASR performance by 8%. To the best of our knowledge, this is also the first work (i) to formulate an alpha smoothing framework to use the non-autoregressive BERT language model for an ASR task, and (ii) to explore sub-word units with Transformer-XL for an agglutinative language like Finnish.

KW - 6121 Languages

KW - 113 Computer and information sciences

U2 - 10.21437/Interspeech.2020-1784

DO - 10.21437/Interspeech.2020-1784

M3 - Conference contribution

T3 - Interspeech

SP - 3630

EP - 3634

BT - Proceedings of Interspeech 2020

PB - ISCA - International Speech Communication Association

CY - Baixas

T2 - Interspeech 2020

Y2 - 25 October 2020 through 29 October 2020

ER -