Abstrakti
This paper studies the correction of challenging authentic Finnish learner texts at beginner level (CEFR A1). Three state-of-the-art large language models are compared, and it is shown that GPT-4 outperforms GPT-3.5, which in turn outperforms Claude v1 on this task. Additionally, ensemble models based on classifiers combining outputs of multiple single models are evaluated. The highest accuracy for an ensemble model is 84.3%, whereas the best single model, which is a GPT-4 model, produces sentences that are fully correct 83.3% of the time. In general, the different models perform on a continuum, where grammatical correctness, fluency and coherence go hand in hand.
Alkuperäiskieli | englanti |
---|---|
Otsikko | Proceedings of the Ninth Workshop on Noisy and User-generated Text (W-NUT 2024) : Collocated with EACL 2024 |
Toimittajat | Rob van der Goot, JinYeong Bak, Max Müller-Eberstein, Wei Xu, Alan Ritter, Tim Baldwin |
Sivumäärä | 10 |
Julkaisupaikka | Stroudsburg |
Kustantaja | Association for Computational Linguistics (ACL) |
Julkaisupäivä | 22 maalisk. 2024 |
Sivut | 1-10 |
ISBN (elektroninen) | 979-8-89176-087-5 |
Tila | Julkaistu - 22 maalisk. 2024 |
OKM-julkaisutyyppi | A4 Artikkeli konferenssijulkaisuussa |
Tapahtuma | Workshop on Noisy and User-generated Text - St. Julian’s, Malta Kesto: 22 maalisk. 2024 → 22 maalisk. 2024 Konferenssinumero: 9 |
Tieteenalat
- 6121 Kielitieteet
- 113 Tietojenkäsittely- ja informaatiotieteet