ELOQUENT CLEF Shared Tasks for Evaluation of Generative Language Model Quality

Jussi Jerker Karlgren; Luise Dürlich; Evangelia Gogoulou; Liane Guillou; Joakim Nivre; Magnus Sahlgren; Aarne Talman

doi:10.1007/978-3-031-56069-9_63

ELOQUENT CLEF Shared Tasks for Evaluation of Generative Language Model Quality

Jussi Jerker Karlgren, Luise Dürlich, Evangelia Gogoulou, Liane Guillou, Joakim Nivre, Magnus Sahlgren, Aarne Talman

Forskningsoutput: Kapitel i bok/rapport/konferenshandling › Konferensbidrag › Vetenskaplig › Peer review

Sammanfattning

ELOQUENT is a set of shared tasks for evaluating the quality and usefulness of generative language models. ELOQUENT aims to bring together some high-level quality criteria, grounded in experiences from deploying models in real-life tasks, and to formulate tests for those criteria, preferably implemented to require minimal human assessment effort and in a multilingual setting. The selected tasks for this first year of ELOQUENT are (1) probing a language model for topical competence; (2) assessing the ability of models to generate and detect hallucinations; (3) assessing the robustness of a model output given variation in the input prompts; and (4) establishing the possibility to distinguish human-generated text from machine-generated text.

Originalspråk	engelska
Titel på värdpublikation	Advances in Information Retrieval. ECIR 2024
Redaktörer	N. Goharian, et al.
Utgivningsort	Cham
Förlag	Springer
Utgivningsdatum	23 mars 2024
Sidor	459–465
ISBN (tryckt)	978-3-031-56068-2
ISBN (elektroniskt)	978-3-031-56069-9
DOI	https://doi.org/10.1007/978-3-031-56069-9_63
Status	Publicerad - 23 mars 2024
Externt publicerad	Ja
MoE-publikationstyp	A4 Artikel i en konferenspublikation
Evenemang	European Conference on Information Retrieval: ECIR - Glasgow, Storbritannien Varaktighet: 24 mars 2024 → 28 mars 2024 Konferensnummer: 46

Publikationsserier

Namn	Lecture Notes in Computer Science
Förlag	Springer
Volym	14612
ISSN (tryckt)	0302-9743
ISSN (elektroniskt)	1611-3349

Vetenskapsgrenar

6121 Språkvetenskaper
113 Data- och informationsvetenskap

Åtkomst av dokument

10.1007/978-3-031-56069-9_63

Citera det här

Karlgren, J. J., Dürlich, L., Gogoulou, E., Guillou, L., Nivre, J., Sahlgren, M., & Talman, A. (2024). ELOQUENT CLEF Shared Tasks for Evaluation of Generative Language Model Quality. I N. Goharian, et al. (Red.), Advances in Information Retrieval. ECIR 2024 (s. 459–465). ( Lecture Notes in Computer Science; Vol. 14612). Springer . https://doi.org/10.1007/978-3-031-56069-9_63

@inproceedings{ec6b180ff3f64b839dee05726cc94027,

title = "ELOQUENT CLEF Shared Tasks for Evaluation of Generative Language Model Quality",

abstract = "ELOQUENT is a set of shared tasks for evaluating the quality and usefulness of generative language models. ELOQUENT aims to bring together some high-level quality criteria, grounded in experiences from deploying models in real-life tasks, and to formulate tests for those criteria, preferably implemented to require minimal human assessment effort and in a multilingual setting. The selected tasks for this first year of ELOQUENT are (1) probing a language model for topical competence; (2) assessing the ability of models to generate and detect hallucinations; (3) assessing the robustness of a model output given variation in the input prompts; and (4) establishing the possibility to distinguish human-generated text from machine-generated text.",

keywords = "6121 Languages, 113 Computer and information sciences",

author = "Karlgren, {Jussi Jerker} and Luise D{\"u}rlich and Evangelia Gogoulou and Liane Guillou and Joakim Nivre and Magnus Sahlgren and Aarne Talman",

year = "2024",

month = mar,

day = "23",

doi = "10.1007/978-3-031-56069-9_63",

language = "English",

isbn = "978-3-031-56068-2",

series = " Lecture Notes in Computer Science",

publisher = "Springer ",

pages = "459–465",

editor = "{Goharian, et al.}, N.",

booktitle = "Advances in Information Retrieval. ECIR 2024",

address = "International",

note = "European Conference on Information Retrieval : ECIR ; Conference date: 24-03-2024 Through 28-03-2024",

}

Karlgren, JJ, Dürlich, L, Gogoulou, E, Guillou, L, Nivre, J, Sahlgren, M & Talman, A 2024, ELOQUENT CLEF Shared Tasks for Evaluation of Generative Language Model Quality. i N Goharian, et al. (red.), Advances in Information Retrieval. ECIR 2024. Lecture Notes in Computer Science, vol. 14612, Springer , Cham, s. 459–465, European Conference on Information Retrieval, Glasgow, Storbritannien, 24/03/2024. https://doi.org/10.1007/978-3-031-56069-9_63

ELOQUENT CLEF Shared Tasks for Evaluation of Generative Language Model Quality. / Karlgren, Jussi Jerker; Dürlich, Luise; Gogoulou, Evangelia et al.
Advances in Information Retrieval. ECIR 2024. red. / N. Goharian, et al. Cham: Springer , 2024. s. 459–465 ( Lecture Notes in Computer Science; Vol. 14612).

Forskningsoutput: Kapitel i bok/rapport/konferenshandling › Konferensbidrag › Vetenskaplig › Peer review

TY - GEN

T1 - ELOQUENT CLEF Shared Tasks for Evaluation of Generative Language Model Quality

AU - Karlgren, Jussi Jerker

AU - Dürlich, Luise

AU - Gogoulou, Evangelia

AU - Guillou, Liane

AU - Nivre, Joakim

AU - Sahlgren, Magnus

AU - Talman, Aarne

N1 - Conference code: 46

PY - 2024/3/23

Y1 - 2024/3/23

N2 - ELOQUENT is a set of shared tasks for evaluating the quality and usefulness of generative language models. ELOQUENT aims to bring together some high-level quality criteria, grounded in experiences from deploying models in real-life tasks, and to formulate tests for those criteria, preferably implemented to require minimal human assessment effort and in a multilingual setting. The selected tasks for this first year of ELOQUENT are (1) probing a language model for topical competence; (2) assessing the ability of models to generate and detect hallucinations; (3) assessing the robustness of a model output given variation in the input prompts; and (4) establishing the possibility to distinguish human-generated text from machine-generated text.

AB - ELOQUENT is a set of shared tasks for evaluating the quality and usefulness of generative language models. ELOQUENT aims to bring together some high-level quality criteria, grounded in experiences from deploying models in real-life tasks, and to formulate tests for those criteria, preferably implemented to require minimal human assessment effort and in a multilingual setting. The selected tasks for this first year of ELOQUENT are (1) probing a language model for topical competence; (2) assessing the ability of models to generate and detect hallucinations; (3) assessing the robustness of a model output given variation in the input prompts; and (4) establishing the possibility to distinguish human-generated text from machine-generated text.

KW - 6121 Languages

KW - 113 Computer and information sciences

U2 - 10.1007/978-3-031-56069-9_63

DO - 10.1007/978-3-031-56069-9_63

M3 - Conference contribution

SN - 978-3-031-56068-2

T3 - Lecture Notes in Computer Science

SP - 459

EP - 465

BT - Advances in Information Retrieval. ECIR 2024

A2 - Goharian, et al., N.

PB - Springer

CY - Cham

T2 - European Conference on Information Retrieval

Y2 - 24 March 2024 through 28 March 2024

ER -