Large expert-curated database for benchmarking document similarity detection in biomedical literature search

RELISH Consortium, Yaoqi Zhou, Jari Yli-Kauhaluoma, Gabriel Magno de Freitas Almeida, Pedro Cardoso, Eeva-Liisa Eskelinen, Heikki Helanterä, Elina Sillanpää, Mikko Frilander, Kalevi Korpela, Carlos Pinto, Gwendolyn Quinn, Sushil Tripathi, Zhen Zeng

Forskningsoutput: TidskriftsbidragArtikelVetenskapligPeer review

Sammanfattning

Document recommendation systems for locating relevant literature have mostly relied on methods developed a decade ago. This is largely due to the lack of a large offline gold-standard benchmark of relevant documents that cover a variety of research fields such that newly developed literature search techniques can be compared, improved and translated into practice. To overcome this bottleneck, we have established the RElevant LIterature SearcH consortium consisting of more than 1500 scientists from 84 countries, who have collectively annotated the relevance of over 180 000 PubMed-listed articles with regard to their respective seed (input) article/s. The majority of annotations were contributed by highly experienced, original authors of the seed articles. The collected data cover 76% of all unique PubMed Medical Subject Headings descriptors. No systematic biases were observed across different experience levels, research fields or time spent on annotations. More importantly, annotations of the same document pairs contributed by different scientists were highly concordant. We further show that the three representative baseline methods used to generate recommended articles for evaluation (Okapi Best Matching 25, Term Frequency-Inverse Document Frequency and PubMed Related Articles) had similar overall performances. Additionally, we found that these methods each tend to produce distinct collections of recommended articles, suggesting that a hybrid method may be required to completely capture all relevant articles. The established database server located at https://relishdb.ict.griffith.edu.au is freely available for the downloading of annotation data and the blind testing of new methods. We expect that this benchmark will be useful for stimulating the development of new powerful techniques for title and title/abstract-based search engines for relevant articles in biomedical research.

Originalspråkengelska
Artikelnummerbaz085
TidskriftDatabase-The journal of biological databases and curation
Volym2019
Sidor (från-till)1-66
Antal sidor66
ISSN1758-0463
DOI
StatusPublicerad - 29 okt 2019
MoE-publikationstypA1 Tidskriftsartikel-refererad

Vetenskapsgrenar

  • 111 Matematik
  • 113 Data- och informationsvetenskap
  • 119 Övrig naturvetenskap
  • 3111 Biomedicinska vetenskaper

Citera det här

RELISH Consortium ; Zhou, Yaoqi ; Yli-Kauhaluoma, Jari ; Almeida, Gabriel Magno de Freitas ; Cardoso, Pedro ; Eskelinen, Eeva-Liisa ; Helanterä, Heikki ; Sillanpää, Elina ; Frilander, Mikko ; Korpela, Kalevi ; Pinto, Carlos ; Quinn, Gwendolyn ; Tripathi, Sushil ; Zeng, Zhen. / Large expert-curated database for benchmarking document similarity detection in biomedical literature search. I: Database-The journal of biological databases and curation. 2019 ; Vol. 2019. s. 1-66.
@article{ffed3f1a85ce41809fadf432f8bd7409,
title = "Large expert-curated database for benchmarking document similarity detection in biomedical literature search",
abstract = "Document recommendation systems for locating relevant literature have mostly relied on methods developed a decade ago. This is largely due to the lack of a large offline gold-standard benchmark of relevant documents that cover a variety of research fields such that newly developed literature search techniques can be compared, improved and translated into practice. To overcome this bottleneck, we have established the RElevant LIterature SearcH consortium consisting of more than 1500 scientists from 84 countries, who have collectively annotated the relevance of over 180 000 PubMed-listed articles with regard to their respective seed (input) article/s. The majority of annotations were contributed by highly experienced, original authors of the seed articles. The collected data cover 76{\%} of all unique PubMed Medical Subject Headings descriptors. No systematic biases were observed across different experience levels, research fields or time spent on annotations. More importantly, annotations of the same document pairs contributed by different scientists were highly concordant. We further show that the three representative baseline methods used to generate recommended articles for evaluation (Okapi Best Matching 25, Term Frequency-Inverse Document Frequency and PubMed Related Articles) had similar overall performances. Additionally, we found that these methods each tend to produce distinct collections of recommended articles, suggesting that a hybrid method may be required to completely capture all relevant articles. The established database server located at https://relishdb.ict.griffith.edu.au is freely available for the downloading of annotation data and the blind testing of new methods. We expect that this benchmark will be useful for stimulating the development of new powerful techniques for title and title/abstract-based search engines for relevant articles in biomedical research.",
keywords = "111 Mathematics, 113 Computer and information sciences, 119 Other natural sciences, 3111 Biomedicine",
author = "Peter Brown and {RELISH Consortium} and Yaoqi Zhou and Jari Yli-Kauhaluoma and Almeida, {Gabriel Magno de Freitas} and Pedro Cardoso and Eeva-Liisa Eskelinen and Heikki Helanter{\"a} and Elina Sillanp{\"a}{\"a} and Mikko Frilander and Kalevi Korpela and Carlos Pinto and Gwendolyn Quinn and Sushil Tripathi and Zhen Zeng",
year = "2019",
month = "10",
day = "29",
doi = "10.1093/database/baz085",
language = "English",
volume = "2019",
pages = "1--66",
journal = "Database-The journal of biological databases and curation",
issn = "1758-0463",
publisher = "Oxford University Press",

}

Large expert-curated database for benchmarking document similarity detection in biomedical literature search. / RELISH Consortium; Zhou, Yaoqi; Yli-Kauhaluoma, Jari; Almeida, Gabriel Magno de Freitas; Cardoso, Pedro; Eskelinen, Eeva-Liisa; Helanterä, Heikki; Sillanpää, Elina; Frilander, Mikko; Korpela, Kalevi; Pinto, Carlos ; Quinn, Gwendolyn; Tripathi, Sushil; Zeng, Zhen.

I: Database-The journal of biological databases and curation, Vol. 2019, baz085, 29.10.2019, s. 1-66.

Forskningsoutput: TidskriftsbidragArtikelVetenskapligPeer review

TY - JOUR

T1 - Large expert-curated database for benchmarking document similarity detection in biomedical literature search

AU - Brown, Peter

AU - RELISH Consortium

AU - Zhou, Yaoqi

AU - Yli-Kauhaluoma, Jari

AU - Almeida, Gabriel Magno de Freitas

AU - Cardoso, Pedro

AU - Eskelinen, Eeva-Liisa

AU - Helanterä, Heikki

AU - Sillanpää, Elina

AU - Frilander, Mikko

AU - Korpela, Kalevi

AU - Pinto, Carlos

AU - Quinn, Gwendolyn

AU - Tripathi, Sushil

AU - Zeng, Zhen

PY - 2019/10/29

Y1 - 2019/10/29

N2 - Document recommendation systems for locating relevant literature have mostly relied on methods developed a decade ago. This is largely due to the lack of a large offline gold-standard benchmark of relevant documents that cover a variety of research fields such that newly developed literature search techniques can be compared, improved and translated into practice. To overcome this bottleneck, we have established the RElevant LIterature SearcH consortium consisting of more than 1500 scientists from 84 countries, who have collectively annotated the relevance of over 180 000 PubMed-listed articles with regard to their respective seed (input) article/s. The majority of annotations were contributed by highly experienced, original authors of the seed articles. The collected data cover 76% of all unique PubMed Medical Subject Headings descriptors. No systematic biases were observed across different experience levels, research fields or time spent on annotations. More importantly, annotations of the same document pairs contributed by different scientists were highly concordant. We further show that the three representative baseline methods used to generate recommended articles for evaluation (Okapi Best Matching 25, Term Frequency-Inverse Document Frequency and PubMed Related Articles) had similar overall performances. Additionally, we found that these methods each tend to produce distinct collections of recommended articles, suggesting that a hybrid method may be required to completely capture all relevant articles. The established database server located at https://relishdb.ict.griffith.edu.au is freely available for the downloading of annotation data and the blind testing of new methods. We expect that this benchmark will be useful for stimulating the development of new powerful techniques for title and title/abstract-based search engines for relevant articles in biomedical research.

AB - Document recommendation systems for locating relevant literature have mostly relied on methods developed a decade ago. This is largely due to the lack of a large offline gold-standard benchmark of relevant documents that cover a variety of research fields such that newly developed literature search techniques can be compared, improved and translated into practice. To overcome this bottleneck, we have established the RElevant LIterature SearcH consortium consisting of more than 1500 scientists from 84 countries, who have collectively annotated the relevance of over 180 000 PubMed-listed articles with regard to their respective seed (input) article/s. The majority of annotations were contributed by highly experienced, original authors of the seed articles. The collected data cover 76% of all unique PubMed Medical Subject Headings descriptors. No systematic biases were observed across different experience levels, research fields or time spent on annotations. More importantly, annotations of the same document pairs contributed by different scientists were highly concordant. We further show that the three representative baseline methods used to generate recommended articles for evaluation (Okapi Best Matching 25, Term Frequency-Inverse Document Frequency and PubMed Related Articles) had similar overall performances. Additionally, we found that these methods each tend to produce distinct collections of recommended articles, suggesting that a hybrid method may be required to completely capture all relevant articles. The established database server located at https://relishdb.ict.griffith.edu.au is freely available for the downloading of annotation data and the blind testing of new methods. We expect that this benchmark will be useful for stimulating the development of new powerful techniques for title and title/abstract-based search engines for relevant articles in biomedical research.

KW - 111 Mathematics

KW - 113 Computer and information sciences

KW - 119 Other natural sciences

KW - 3111 Biomedicine

U2 - 10.1093/database/baz085

DO - 10.1093/database/baz085

M3 - Article

VL - 2019

SP - 1

EP - 66

JO - Database-The journal of biological databases and curation

JF - Database-The journal of biological databases and curation

SN - 1758-0463

M1 - baz085

ER -