Abstract
This paper studies the correction of challenging authentic Finnish learner texts at beginner level (CEFR A1). Three state-of-the-art large language models are compared, and it is shown that GPT-4 outperforms GPT-3.5, which in turn outperforms Claude v1 on this task. Additionally, ensemble models based on classifiers combining outputs of multiple single models are evaluated. The highest accuracy for an ensemble model is 84.3%, whereas the best single model, which is a GPT-4 model, produces sentences that are fully correct 83.3% of the time. In general, the different models perform on a continuum, where grammatical correctness, fluency and coherence go hand in hand.
| Original language | English |
|---|---|
| Title of host publication | Proceedings of the Ninth Workshop on Noisy and User-generated Text (W-NUT 2024) : Collocated with EACL 2024 |
| Editors | Rob van der Goot, JinYeong Bak, Max Müller-Eberstein, Wei Xu, Alan Ritter, Tim Baldwin |
| Number of pages | 10 |
| Place of Publication | Stroudsburg |
| Publisher | Association for Computational Linguistics (ACL) |
| Publication date | 22 Mar 2024 |
| Pages | 1-10 |
| ISBN (Electronic) | 979-8-89176-087-5 |
| Publication status | Published - 22 Mar 2024 |
| MoE publication type | A4 Article in conference proceedings |
| Event | Workshop on Noisy and User-generated Text - St. Julian’s, Malta Duration: 22 Mar 2024 → 22 Mar 2024 Conference number: 9 |
Fields of Science
- 6121 Languages
- 113 Computer and information sciences