Robust Multimodal Representation Learning with Evolutionary Adversarial Attention Networks

Feiran Huang, Alireza Jolfaei, Ali Kashif Bashir

Research output: Contribution to journalArticlepeer-review

2 Citations (Scopus)

Abstract

Multimodal representation learning is beneficial for many multimedia-oriented applications, such as social image recognition and visual question answering. The different modalities of the same instance (e.g., a social image and its corresponding description) are usually correlational and complementary. Most existing approaches for multimodal representation learning are not effective to model the deep correlation between different modalities. Moreover, it is difficult for these approaches to deal with the noise within social images. In this article, we propose a deep learning-based approach named evolutionary adversarial attention networks (EAANs), which combines the attention mechanism with adversarial networks through evolutionary training, for robust multimodal representation learning. Specifically, a two-branch visual-textual attention model is proposed to correlate visual and textual content for joint representation. Then adversarial networks are employed to impose regularization upon the representation by matching its posterior distribution to the given priors. Finally, the attention model and adversarial networks are integrated into an evolutionary training framework for robust multimodal representation learning. Extensive experiments have been conducted on four real-world datasets, including PASCAL, MIR, CLEF, and NUS-WIDE. Substantial performance improvements on the tasks of image classification and tag recommendation demonstrate the superiority of the proposed approach.

Original languageEnglish
Pages (from-to)856-868
Number of pages13
JournalIEEE Transactions on Evolutionary Computation
Volume25
Issue number5
DOIs
Publication statusPublished - 1 Oct 2021
Externally publishedYes

Keywords

  • Adversarial networks
  • attention model
  • evolutionary
  • multimodal
  • representation learning

Fingerprint

Dive into the research topics of 'Robust Multimodal Representation Learning with Evolutionary Adversarial Attention Networks'. Together they form a unique fingerprint.

Cite this