SemEval-2024 Task 6: SHROOM, a Shared-task on Hallucinations and Related Observable Overgeneration Mistakes

Timothee Mickus, Elaine Zosa, Raul Vazquez, Teemu Vahtola, Jörg Tiedemann, Vincent Segonne, Alessandro Raganato, Marianna Apidianaki

Research output: Chapter in Book/Report/Conference proceedingConference contributionScientificpeer-review

Abstract

This paper presents the results of the SHROOM, a shared task focused on detecting hallucinations: outputs from natural language generation (NLG) systems that are fluent, yet inaccurate. Such cases of overgeneration put in jeopardy many NLG applications, where correctness is often mission-critical. The shared task was conducted with a newly constructed dataset of 4000 model outputs labeled by 5 annotators each, spanning 3 NLP tasks: machine translation, paraphrase generation and definition modeling.The shared task was tackled by a total of 58 different users grouped in 42 teams, out of which 26 elected to write a system description paper; collectively, they submitted over 300 prediction sets on both tracks of the shared task. We observe a number of key trends in how this approach was tackled---many participants rely on a handful of model, and often rely either on synthetic data for fine-tuning or zero-shot prompting strategies. While a majority of the teams did outperform our proposed baseline system, the performances of top-scoring systems are still consistent with a random handling of the more challenging items.
Original languageEnglish
Title of host publicationProceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024)
EditorsAtul Kr. Ojha, A. Seza Doğruöz, Harish Tayyar Madabushi, Giovanni Da San Martino, Sara Rosenthal, Aiala Rosá
Number of pages15
Place of PublicationStroudsburg
PublisherThe Association for Computational Linguistics
Publication date1 Jun 2024
Pages1979-1993
ISBN (Electronic)979-8-89176-107-0
Publication statusPublished - 1 Jun 2024
MoE publication typeA4 Article in conference proceedings
EventInternational Workshop on Semantic Evaluation - Mexico City, Mexico
Duration: 20 Jun 202421 Jun 2024
Conference number: 18

Fields of Science

  • 6121 Languages
  • 113 Computer and information sciences

Cite this