Sentiment-aware Classifier for Out-of-Context Caption Detection

Muhannad Alkaddour, Abhinav Dhall, Usman Tariq, Hasan Al-Nashash, Fares Al-Shargie

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

4 Citations (Scopus)

Abstract

In this work we propose additions to the COSMOS and COSMOS on Steroids pipelines for the detection of Cheapfakes for Task 1 of the ACM Grand Challenge for Detecting Cheapfakes. We compute sentiment features, namely polarity and subjectivity, using the news image captions. Multiple logistic regression results show that these sentiment features are significant in prediction of the outcome. We then combine the sentiment features with the four image-text features obtained in the aforementioned previous works to train an MLP. This classifies sets of inputs into being out-of-context (OOC) or not-out-of-context (NOOC). On a test set of 400 samples, the MLP with all features achieved a score of 87.25%, and that with only the image-text features a score of 88%. In addition to the challenge requirements, we also propose a separate pipeline to automatically construct caption pairs and annotations using the images and captions provided in the large, un-annotated training dataset. We hope that this endeavor will open the door for improvements, since hand-annotating cheapfake labels is time-consuming. To evaluate the performance on the test set, the Docker image with the models is available at: https://hub.docker.com/repository/docker/malkaddour/mmsys22cheapfakes. The open-source code for the project is accessible at: https://github.com/malkaddour/ACMM-22-Cheapfake-Detection-Sentiment-aware-Classifier-for-Out-of-Context-Caption-Detection.

Original languageEnglish
Title of host publicationMM '22
Subtitle of host publicationProceedings of the 30th ACM International Conference on Multimedia
PublisherAssociation for Computing Machinery, Inc
Pages7180-7184
Number of pages5
ISBN (Electronic)9781450392037
DOIs
Publication statusPublished - 10 Oct 2022
Externally publishedYes
Event30th ACM International Conference on Multimedia - Lisbon, Portugal
Duration: 10 Oct 202214 Oct 2022
Conference number: 30th

Conference

Conference30th ACM International Conference on Multimedia
Abbreviated titleMM '22
Country/TerritoryPortugal
CityLisbon
Period10/10/2214/10/22

Keywords

  • cheapfake detection
  • cosmos
  • nlp
  • sentiment analysis

Fingerprint

Dive into the research topics of 'Sentiment-aware Classifier for Out-of-Context Caption Detection'. Together they form a unique fingerprint.

Cite this