Not made for each other: Audio-Visual Dissonance-based Deepfake Detection and Localization

Komal Chugh; Parul Gupta; Abhinav Dhall; Ramanathan Subramanian

doi:10.1145/3394171.3413700

Not made for each other: Audio-Visual Dissonance-based Deepfake Detection and Localization

Komal Chugh, Parul Gupta, Abhinav Dhall, Ramanathan Subramanian

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

186 Citations (Scopus)

Abstract

We propose detection of deepfake videos based on the dissimilarity between the audio and visual modalities, termed as the Modality Dissonance Score (MDS). We hypothesize that manipulation of either modality will lead to dis-harmony between the two modalities, e.g., loss of lip-sync, unnatural facial and lip movements, etc. MDS is computed as the mean aggregate of dissimilarity scores between audio and visual segments in a video. Discriminative features are learnt for the audio and visual channels in a chunk-wise manner, employing the cross-entropy loss for individual modalities, and a contrastive loss that models inter-modality similarity. Extensive experiments on the DFDC and DeepFake-TIMIT Datasets show that our approach outperforms the state-of-the-art by up to 7%. We also demonstrate temporal forgery localization, and show how our technique identifies the manipulated video segments.

Original language	English
Title of host publication	MM 2020 - Proceedings of the 28th ACM International Conference on Multimedia
Publisher	Association for Computing Machinery, Inc
Pages	439-447
Number of pages	9
ISBN (Electronic)	9781450379885
DOIs	https://doi.org/10.1145/3394171.3413700
Publication status	Published - 12 Oct 2020
Externally published	Yes
Event	28th ACM International Conference on Multimedia - Virtual, Online, United States Duration: 12 Oct 2020 → 16 Oct 2020 Conference number: 28th

Publication series

Name	MM 2020 - Proceedings of the 28th ACM International Conference on Multimedia

Conference

Conference	28th ACM International Conference on Multimedia
Abbreviated title	MM '20
Country/Territory	United States
City	Virtual, Online
Period	12/10/20 → 16/10/20
Other	This year's conference, for the first time in the history, will be running as an online large-scale meeting due to COVID-19. Face-to-face gatherings have been replaced by all-hands live sessions, parallel live question-answering (Q&A) sessions, and presentations from the authors via pre-recorded videos. However, the distance needed to prevent the spreading of the virus does not cool down enthusiasm in participants, presenters, chairs, reviewers, and volunteers. Under today’s challenging circumstance, we are pleased that we are still able to deliver the most cutting-edge research results, covering the latest findings in the field, for which the ACM Multimedia conference series is widely known.

Keywords

contrastive loss
deepfake detection and localization
modality dissonance
neural networks

Access to Document

10.1145/3394171.3413700Licence: In Copyright

Cite this

Chugh, K., Gupta, P., Dhall, A., & Subramanian, R. (2020). Not made for each other: Audio-Visual Dissonance-based Deepfake Detection and Localization. In MM 2020 - Proceedings of the 28th ACM International Conference on Multimedia (pp. 439-447). (MM 2020 - Proceedings of the 28th ACM International Conference on Multimedia). Association for Computing Machinery, Inc. https://doi.org/10.1145/3394171.3413700

@inproceedings{26490a66b453405885fdc5d522e6796f,

title = "Not made for each other: Audio-Visual Dissonance-based Deepfake Detection and Localization",

abstract = "We propose detection of deepfake videos based on the dissimilarity between the audio and visual modalities, termed as the Modality Dissonance Score (MDS). We hypothesize that manipulation of either modality will lead to dis-harmony between the two modalities, e.g., loss of lip-sync, unnatural facial and lip movements, etc. MDS is computed as the mean aggregate of dissimilarity scores between audio and visual segments in a video. Discriminative features are learnt for the audio and visual channels in a chunk-wise manner, employing the cross-entropy loss for individual modalities, and a contrastive loss that models inter-modality similarity. Extensive experiments on the DFDC and DeepFake-TIMIT Datasets show that our approach outperforms the state-of-the-art by up to 7%. We also demonstrate temporal forgery localization, and show how our technique identifies the manipulated video segments.",

keywords = "contrastive loss, deepfake detection and localization, modality dissonance, neural networks",

author = "Komal Chugh and Parul Gupta and Abhinav Dhall and Ramanathan Subramanian",

year = "2020",

month = oct,

day = "12",

doi = "10.1145/3394171.3413700",

language = "English",

series = "MM 2020 - Proceedings of the 28th ACM International Conference on Multimedia",

publisher = "Association for Computing Machinery, Inc",

pages = "439--447",

booktitle = "MM 2020 - Proceedings of the 28th ACM International Conference on Multimedia",

note = "28th ACM International Conference on Multimedia, MM '20 ; Conference date: 12-10-2020 Through 16-10-2020",

}

Chugh, K, Gupta, P, Dhall, A & Subramanian, R 2020, Not made for each other: Audio-Visual Dissonance-based Deepfake Detection and Localization. in MM 2020 - Proceedings of the 28th ACM International Conference on Multimedia. MM 2020 - Proceedings of the 28th ACM International Conference on Multimedia, Association for Computing Machinery, Inc, pp. 439-447, 28th ACM International Conference on Multimedia, Virtual, Online, United States, 12/10/20. https://doi.org/10.1145/3394171.3413700

Not made for each other: Audio-Visual Dissonance-based Deepfake Detection and Localization. / Chugh, Komal; Gupta, Parul; Dhall, Abhinav et al.
MM 2020 - Proceedings of the 28th ACM International Conference on Multimedia. Association for Computing Machinery, Inc, 2020. p. 439-447 (MM 2020 - Proceedings of the 28th ACM International Conference on Multimedia).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Not made for each other

T2 - 28th ACM International Conference on Multimedia

AU - Chugh, Komal

AU - Gupta, Parul

AU - Dhall, Abhinav

AU - Subramanian, Ramanathan

N1 - Conference code: 28th

PY - 2020/10/12

Y1 - 2020/10/12

N2 - We propose detection of deepfake videos based on the dissimilarity between the audio and visual modalities, termed as the Modality Dissonance Score (MDS). We hypothesize that manipulation of either modality will lead to dis-harmony between the two modalities, e.g., loss of lip-sync, unnatural facial and lip movements, etc. MDS is computed as the mean aggregate of dissimilarity scores between audio and visual segments in a video. Discriminative features are learnt for the audio and visual channels in a chunk-wise manner, employing the cross-entropy loss for individual modalities, and a contrastive loss that models inter-modality similarity. Extensive experiments on the DFDC and DeepFake-TIMIT Datasets show that our approach outperforms the state-of-the-art by up to 7%. We also demonstrate temporal forgery localization, and show how our technique identifies the manipulated video segments.

AB - We propose detection of deepfake videos based on the dissimilarity between the audio and visual modalities, termed as the Modality Dissonance Score (MDS). We hypothesize that manipulation of either modality will lead to dis-harmony between the two modalities, e.g., loss of lip-sync, unnatural facial and lip movements, etc. MDS is computed as the mean aggregate of dissimilarity scores between audio and visual segments in a video. Discriminative features are learnt for the audio and visual channels in a chunk-wise manner, employing the cross-entropy loss for individual modalities, and a contrastive loss that models inter-modality similarity. Extensive experiments on the DFDC and DeepFake-TIMIT Datasets show that our approach outperforms the state-of-the-art by up to 7%. We also demonstrate temporal forgery localization, and show how our technique identifies the manipulated video segments.

KW - contrastive loss

KW - deepfake detection and localization

KW - modality dissonance

KW - neural networks

UR - https://dl.acm.org/action/showFmPdf?doi=10.1145%2F3394171

UR - https://dl.acm.org/action/showBmPdf?doi=10.1145%2F3394171

UR - http://www.scopus.com/inward/record.url?scp=85101205270&partnerID=8YFLogxK

U2 - 10.1145/3394171.3413700

DO - 10.1145/3394171.3413700

M3 - Conference contribution

AN - SCOPUS:85101205270

T3 - MM 2020 - Proceedings of the 28th ACM International Conference on Multimedia

SP - 439

EP - 447

BT - MM 2020 - Proceedings of the 28th ACM International Conference on Multimedia

PB - Association for Computing Machinery, Inc

Y2 - 12 October 2020 through 16 October 2020

ER -

Chugh K, Gupta P, Dhall A, Subramanian R. Not made for each other: Audio-Visual Dissonance-based Deepfake Detection and Localization. In MM 2020 - Proceedings of the 28th ACM International Conference on Multimedia. Association for Computing Machinery, Inc. 2020. p. 439-447. (MM 2020 - Proceedings of the 28th ACM International Conference on Multimedia). doi: 10.1145/3394171.3413700

Not made for each other: Audio-Visual Dissonance-based Deepfake Detection and Localization

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this