Domain-specific or Uncertainty-aware models: Does it really make a difference for biomedical text classification?

Aman Sinha; Timothee Mickus; Marianne Clausel; Mathieu Constant; Xavier Coubez

Domain-specific or Uncertainty-aware models: Does it really make a difference for biomedical text classification?

Aman Sinha, Timothee Mickus, Marianne Clausel, Mathieu Constant, Xavier Coubez

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Scientific › peer-review

Abstract

The success of pretrained language models (PLMs) across a spate of use-cases has led to significant investment from the NLP community towards building domain-specific foundational models. On the other hand, in mission critical settings such as biomedical applications, other aspects also factor in---chief of which is a model's ability to produce reasonable estimates of its own uncertainty. In the present study, we discuss these two desiderata through the lens of how they shape the entropy of a model's output probability distribution. We find that domain specificity and uncertainty awareness can often be successfully combined, but the exact task at hand weighs in much more strongly.

Original language	English
Title of host publication	Proceedings of the 23rd Workshop on Biomedical Natural Language Processing
Editors	Dina Demner-Fushman, Sophia Ananiadou, Makoto Miwa, Kirk Roberts, Junichi Tsujii
Number of pages	10
Place of Publication	Stroudsburg
Publisher	The Association for Computational Linguistics
Publication date	1 Aug 2024
Pages	202-211
ISBN (Electronic)	979-8-89176-130-8
Publication status	Published - 1 Aug 2024
MoE publication type	A4 Article in conference proceedings
Event	Workshop on Biomedical Natural Language Processing - Bangkok, Thailand Duration: 16 Aug 2024 → 16 Aug 2024 Conference number: 23

Fields of Science

6121 Languages
113 Computer and information sciences

Access to Document

2024.bionlp-1.16Final published version, 857 KBLicence: CC BY

https://aclanthology.org/2024.bionlp-1.16Licence: CC BY

Uncertainty-aware neural language models
Tiedemann, J. (Project manager), Celikkanat, H. (Participant), Virpioja, S. P. (Participant) & Vazquez , R. (Participant)
Academy of Finland, Suomen Akatemia Projektilaskutus
01/01/2022 → 01/10/2025
Project: Research project

Cite this

Sinha, A., Mickus, T., Clausel, M., Constant, M., & Coubez, X. (2024). Domain-specific or Uncertainty-aware models: Does it really make a difference for biomedical text classification? In D. Demner-Fushman, S. Ananiadou, M. Miwa, K. Roberts, & J. Tsujii (Eds.), Proceedings of the 23rd Workshop on Biomedical Natural Language Processing (pp. 202-211). The Association for Computational Linguistics. https://aclanthology.org/2024.bionlp-1.16

Sinha, Aman ; Mickus, Timothee ; Clausel, Marianne et al. / Domain-specific or Uncertainty-aware models : Does it really make a difference for biomedical text classification?. Proceedings of the 23rd Workshop on Biomedical Natural Language Processing. editor / Dina Demner-Fushman ; Sophia Ananiadou ; Makoto Miwa ; Kirk Roberts ; Junichi Tsujii. Stroudsburg : The Association for Computational Linguistics, 2024. pp. 202-211

@inproceedings{c16321ed8a2449d0b6c9d2efe2f96caa,

title = "Domain-specific or Uncertainty-aware models: Does it really make a difference for biomedical text classification?",

abstract = "The success of pretrained language models (PLMs) across a spate of use-cases has led to significant investment from the NLP community towards building domain-specific foundational models. On the other hand, in mission critical settings such as biomedical applications, other aspects also factor in---chief of which is a model's ability to produce reasonable estimates of its own uncertainty. In the present study, we discuss these two desiderata through the lens of how they shape the entropy of a model's output probability distribution. We find that domain specificity and uncertainty awareness can often be successfully combined, but the exact task at hand weighs in much more strongly.",

keywords = "6121 Languages, 113 Computer and information sciences",

author = "Aman Sinha and Timothee Mickus and Marianne Clausel and Mathieu Constant and Xavier Coubez",

year = "2024",

month = aug,

day = "1",

language = "English",

pages = "202--211",

editor = "Dina Demner-Fushman and Sophia Ananiadou and Makoto Miwa and Kirk Roberts and Junichi Tsujii",

booktitle = "Proceedings of the 23rd Workshop on Biomedical Natural Language Processing",

publisher = "The Association for Computational Linguistics",

address = "United States",

note = "Workshop on Biomedical Natural Language Processing ; Conference date: 16-08-2024 Through 16-08-2024",

}

Sinha, A, Mickus, T, Clausel, M, Constant, M & Coubez, X 2024, Domain-specific or Uncertainty-aware models: Does it really make a difference for biomedical text classification? in D Demner-Fushman, S Ananiadou, M Miwa, K Roberts & J Tsujii (eds), Proceedings of the 23rd Workshop on Biomedical Natural Language Processing. The Association for Computational Linguistics, Stroudsburg, pp. 202-211, Workshop on Biomedical Natural Language Processing, Bangkok, Thailand, 16/08/2024. <https://aclanthology.org/2024.bionlp-1.16>

Domain-specific or Uncertainty-aware models: Does it really make a difference for biomedical text classification? / Sinha, Aman; Mickus, Timothee; Clausel, Marianne et al.
Proceedings of the 23rd Workshop on Biomedical Natural Language Processing. ed. / Dina Demner-Fushman; Sophia Ananiadou; Makoto Miwa; Kirk Roberts; Junichi Tsujii. Stroudsburg: The Association for Computational Linguistics, 2024. p. 202-211.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Scientific › peer-review

TY - GEN

T1 - Domain-specific or Uncertainty-aware models

T2 - Workshop on Biomedical Natural Language Processing

AU - Sinha, Aman

AU - Mickus, Timothee

AU - Clausel, Marianne

AU - Constant, Mathieu

AU - Coubez, Xavier

N1 - Conference code: 23

PY - 2024/8/1

Y1 - 2024/8/1

N2 - The success of pretrained language models (PLMs) across a spate of use-cases has led to significant investment from the NLP community towards building domain-specific foundational models. On the other hand, in mission critical settings such as biomedical applications, other aspects also factor in---chief of which is a model's ability to produce reasonable estimates of its own uncertainty. In the present study, we discuss these two desiderata through the lens of how they shape the entropy of a model's output probability distribution. We find that domain specificity and uncertainty awareness can often be successfully combined, but the exact task at hand weighs in much more strongly.

AB - The success of pretrained language models (PLMs) across a spate of use-cases has led to significant investment from the NLP community towards building domain-specific foundational models. On the other hand, in mission critical settings such as biomedical applications, other aspects also factor in---chief of which is a model's ability to produce reasonable estimates of its own uncertainty. In the present study, we discuss these two desiderata through the lens of how they shape the entropy of a model's output probability distribution. We find that domain specificity and uncertainty awareness can often be successfully combined, but the exact task at hand weighs in much more strongly.

KW - 6121 Languages

KW - 113 Computer and information sciences

M3 - Conference contribution

SP - 202

EP - 211

BT - Proceedings of the 23rd Workshop on Biomedical Natural Language Processing

A2 - Demner-Fushman, Dina

A2 - Ananiadou, Sophia

A2 - Miwa, Makoto

A2 - Roberts, Kirk

A2 - Tsujii, Junichi

PB - The Association for Computational Linguistics

CY - Stroudsburg

Y2 - 16 August 2024 through 16 August 2024

ER -

Sinha A, Mickus T, Clausel M, Constant M, Coubez X. Domain-specific or Uncertainty-aware models: Does it really make a difference for biomedical text classification? In Demner-Fushman D, Ananiadou S, Miwa M, Roberts K, Tsujii J, editors, Proceedings of the 23rd Workshop on Biomedical Natural Language Processing. Stroudsburg: The Association for Computational Linguistics. 2024. p. 202-211

Domain-specific or Uncertainty-aware models: Does it really make a difference for biomedical text classification?

Abstract

Fields of Science

Access to Document

Projects

Uncertainty-aware neural language models

Cite this