Do you see what I see? Measuring the semantic differences in image-recognition services' outputs

Research output: Contribution to journalArticleScientificpeer-review

Abstract

As scholars increasingly undertake large-scale analysis of visual materials, advanced computational tools show promise for informing that process. One technique in the toolbox is image recognition, made readily accessible via Google Vision AI, Microsoft Azure Computer Vision, and Amazon's Rekognition service. However, concerns about such issues as bias factors and low reliability have led to warnings against research employing it. A systematic study of cross-service label agreement concretized such issues: using eight datasets, spanning professionally produced and user-generated images, the work showed that image-recognition services disagree on the most suitable labels for images. Beyond supporting caveats expressed in prior literature, the report articulates two mitigation strategies, both involving the use of multiple image-recognition services: Highly explorative research could include all the labels, accepting noisier but less restrictive analysis output. Alternatively, scholars may employ word-embedding-based approaches to identify concepts that are similar enough for their purposes, then focus on those labels filtered in.
Translated title of the contributionNäetkö saman kuin minä? Kuvatunnistuspalvelujen tuotosten semanttisten erojen mittaaminen.
Original languageEnglish
Article number2330-1643
JournalJournal of the Association for Information Science and Technology
Number of pages18
ISSN2330-1635
DOIs
Publication statusPublished - 5 Sept 2023
MoE publication typeA1 Journal article-refereed

Fields of Science

  • 518 Media and communications
  • 113 Computer and information sciences

Cite this