CaptainA self-study mobile app for practising speaking: task completion assessment and feedback with generative AI

Nhan Phan, Anna von Zansen, Maria Kautonen, Tamás Grósz, Mikko Kurimo

Forskningsoutput: Kapitel i bok/rapport/konferenshandlingKonferensbidragVetenskapligPeer review

Sammanfattning

We introduce the CaptainA mobile app, designed to meet the needs of second language (L2) learners engaged in self-study of Finnish, with potential applicability to other languages. Our app can provide automatic speaking assessment (ASA) of task completion in picture-based tasks, along with grading explanations and corrective feedback. It can also automatically generate pictures for visual tasks, providing users with unlimited practice opportunities. The mobile app is based on ourframework that combines visual natural language generation (NLG), automatic speech recognition (ASR), and prompting
large language model (LLM) for low-resource language. Our goal is to promote the development of next-generation speech-based computer-assisted language learning (CALL) systems capable of providing automatic scoring with feedback for learners, even when minimal speech data of L2 learners is available. While the mobile app demonstration is designed for Finnish, the app can also be tested in English.
Originalspråkengelska
Titel på värdpublikationInterspeech 2024
Antal sidor2
FörlagISCA - International Speech Communication Association
Utgivningsdatum2024
Sidor5212-5213
DOI
StatusPublicerad - 2024
MoE-publikationstypA4 Artikel i en konferenspublikation
EvenemangInterspeech - Kos, Grekland
Varaktighet: 1 sep. 20245 sep. 2024
Konferensnummer: 25
https://interspeech2024.org/

Publikationsserier

NamnInterspeech
FörlagISCA
ISSN (elektroniskt)2958-1796

Vetenskapsgrenar

  • 516 Pedagogik

Citera det här