ELOQUENT 2024 — Robustness Task

Magnus Sahlgren, Jussi Jerker Karlgren, Luise Dürlich, Evangelia Gogoulou, Aarne Talman, Shorouq Zahra

Tutkimustuotos: Artikkeli kirjassa/raportissa/konferenssijulkaisussaKonferenssiartikkeliTieteellinenvertaisarvioitu

Abstrakti

ELOQUENT is a set of shared tasks for evaluating the quality and usefulness of generative language models. ELOQUENT aims to apply high-level quality criteria, grounded in experiences from deploying models in real-life tasks, and to formulate tests for those criteria, preferably implemented to require minimal human assessment effort and in a multilingual setting. One of the tasks for the first year of ELOQUENT was the robustness task, in which we assessed the robustness and consistency of a model output given variation in the input prompts. We found that indeed the consistency varied, both across prompt items and across models, and on a methodological note we find that using a oracle model for assessing the submitted responses is feasible, and intend to investigate consistency across such assessments for different oracle models. We intend to run this task in coming editions for ELOQUENT to establish a solid methodology for further assessing consistency, which we believe to be a crucial component of trustworthiness as a top level quality characteristic of generative language models.

Alkuperäiskielienglanti
OtsikkoWorking Notes of the Conference and Labs of the Evaluation Forum (CLEF 2024)
ToimittajatGuglielmo Faggioli, Nicola Ferro, Petra Galuščáková, Alba García Seco de Herrera
Sivumäärä5
JulkaisupaikkaAachen
KustantajaCEUR-WS.org
Julkaisupäivä2024
Sivut703-707
TilaJulkaistu - 2024
OKM-julkaisutyyppiA4 Artikkeli konferenssijulkaisuussa
TapahtumaConference and Labs of the Evaluation Forum - Grenoble, Ranska
Kesto: 9 syysk. 202412 syysk. 2024
Konferenssinumero: 15

Julkaisusarja

NimiCEUR Workshop Proceedings
Kustantaja CEUR-WS.org
Vuosikerta3740
ISSN (painettu)1613-0073

Lisätietoja

Publisher Copyright:
© 2024 Copyright for this paper by its authors.

Tieteenalat

  • 6121 Kielitieteet
  • 113 Tietojenkäsittely- ja informaatiotieteet

Siteeraa tätä