Projects per year
Abstract
This paper presents the results of the SHROOM, a shared task focused on detecting hallucinations: outputs from natural language generation (NLG) systems that are fluent, yet inaccurate. Such cases of overgeneration put in jeopardy many NLG applications, where correctness is often mission-critical. The shared task was conducted with a newly constructed dataset of 4000 model outputs labeled by 5 annotators each, spanning 3 NLP tasks: machine translation, paraphrase generation and definition modeling.The shared task was tackled by a total of 58 different users grouped in 42 teams, out of which 26 elected to write a system description paper; collectively, they submitted over 300 prediction sets on both tracks of the shared task. We observe a number of key trends in how this approach was tackled---many participants rely on a handful of model, and often rely either on synthetic data for fine-tuning or zero-shot prompting strategies. While a majority of the teams did outperform our proposed baseline system, the performances of top-scoring systems are still consistent with a random handling of the more challenging items.
Original language | English |
---|---|
Title of host publication | Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024) |
Editors | Atul Kr. Ojha, A. Seza Doğruöz, Harish Tayyar Madabushi, Giovanni Da San Martino, Sara Rosenthal, Aiala Rosá |
Number of pages | 15 |
Place of Publication | Stroudsburg |
Publisher | The Association for Computational Linguistics |
Publication date | 1 Jun 2024 |
Pages | 1979-1993 |
ISBN (Electronic) | 979-8-89176-107-0 |
Publication status | Published - 1 Jun 2024 |
MoE publication type | A4 Article in conference proceedings |
Event | International Workshop on Semantic Evaluation - Mexico City, Mexico Duration: 20 Jun 2024 → 21 Jun 2024 Conference number: 18 |
Fields of Science
- 6121 Languages
- 113 Computer and information sciences
Projects
- 1 Active
-
Uncertainty-aware neural language models
Tiedemann, J. (Project manager), Celikkanat, H. (Participant), Virpioja, S. P. (Participant) & Vazquez , R. (Participant)
Academy of Finland, Suomen Akatemia Projektilaskutus
01/01/2022 → 01/10/2025
Project: Research project