MAGIC-TBR: Multiview Attention Fusion for Transformer-based Bodily Behavior Recognition in Group Settings

Surbhi Madan; Rishabh Jain; Gulshan Sharma; Ramanathan Subramanian; Abhinav Dhall

doi:10.1145/3581783.3612858

MAGIC-TBR: Multiview Attention Fusion for Transformer-based Bodily Behavior Recognition in Group Settings

Surbhi Madan, Rishabh Jain, Gulshan Sharma, Ramanathan Subramanian, Abhinav Dhall

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

1 Citation (Scopus)

Abstract

Bodily behavioral language is an important social cue, and its automated analysis helps in enhancing the understanding of artificial intelligence systems. Furthermore, behavioral language cues are essential for active engagement in social agent-based user interactions. Despite the progress made in computer vision for tasks like head and body pose estimation, there is still a need to explore the detection of finer behaviors such as gesturing, grooming, or fumbling. This paper proposes a multiview attention fusion method named MAGIC-TBR that combines features extracted from videos and their corresponding Discrete Cosine Transform coefficients via a transformer-based approach. The experiments are conducted on the BBSI dataset and the results demonstrate the effectiveness of the proposed feature fusion with multiview attention. The code is available at: https://github.com/surbhimadan92/MAGIC-TBR

Original language	English
Title of host publication	MM '23 - Proceedings of the 31st ACM International Conference on Multimedia
Publisher	Association for Computing Machinery, Inc
Pages	9526-9530
Number of pages	5
ISBN (Electronic)	9798400701085
DOIs	https://doi.org/10.1145/3581783.3612858
Publication status	Published - 27 Oct 2023
Externally published	Yes
Event	31st ACM International Conference on Multimedia - Ottawa, Canada Duration: 29 Oct 2023 → 3 Nov 2023 Conference number: 31st

Publication series

Name	Proceedings of the ACM International Conference on Multimedia
Publisher	Association for Computing Machinery, Inc
Number	31st

Conference

Conference	31st ACM International Conference on Multimedia
Abbreviated title	MM '23
Country/Territory	Canada
City	Ottawa
Period	29/10/23 → 3/11/23

Keywords

bodily behavior
dct
multiview attention
transformer

Access to Document

10.1145/3581783.3612858Licence: In Copyright

Cite this

Madan, S., Jain, R., Sharma, G., Subramanian, R., & Dhall, A. (2023). MAGIC-TBR: Multiview Attention Fusion for Transformer-based Bodily Behavior Recognition in Group Settings. In MM '23 - Proceedings of the 31st ACM International Conference on Multimedia (pp. 9526-9530). (Proceedings of the ACM International Conference on Multimedia; No. 31st). Association for Computing Machinery, Inc. https://doi.org/10.1145/3581783.3612858

Madan, Surbhi ; Jain, Rishabh ; Sharma, Gulshan et al. / MAGIC-TBR : Multiview Attention Fusion for Transformer-based Bodily Behavior Recognition in Group Settings. MM '23 - Proceedings of the 31st ACM International Conference on Multimedia. Association for Computing Machinery, Inc, 2023. pp. 9526-9530 (Proceedings of the ACM International Conference on Multimedia; 31st).

@inproceedings{ad1b898a2811404fa27e13d9a148d3fe,

title = "MAGIC-TBR: Multiview Attention Fusion for Transformer-based Bodily Behavior Recognition in Group Settings",

abstract = "Bodily behavioral language is an important social cue, and its automated analysis helps in enhancing the understanding of artificial intelligence systems. Furthermore, behavioral language cues are essential for active engagement in social agent-based user interactions. Despite the progress made in computer vision for tasks like head and body pose estimation, there is still a need to explore the detection of finer behaviors such as gesturing, grooming, or fumbling. This paper proposes a multiview attention fusion method named MAGIC-TBR that combines features extracted from videos and their corresponding Discrete Cosine Transform coefficients via a transformer-based approach. The experiments are conducted on the BBSI dataset and the results demonstrate the effectiveness of the proposed feature fusion with multiview attention. The code is available at: https://github.com/surbhimadan92/MAGIC-TBR",

keywords = "bodily behavior, dct, multiview attention, transformer",

author = "Surbhi Madan and Rishabh Jain and Gulshan Sharma and Ramanathan Subramanian and Abhinav Dhall",

year = "2023",

month = oct,

day = "27",

doi = "10.1145/3581783.3612858",

language = "English",

series = "Proceedings of the ACM International Conference on Multimedia",

publisher = "Association for Computing Machinery, Inc",

number = "31st",

pages = "9526--9530",

booktitle = "MM '23 - Proceedings of the 31st ACM International Conference on Multimedia",

note = "31st ACM International Conference on Multimedia, MM '23 ; Conference date: 29-10-2023 Through 03-11-2023",

}

Madan, S, Jain, R, Sharma, G, Subramanian, R & Dhall, A 2023, MAGIC-TBR: Multiview Attention Fusion for Transformer-based Bodily Behavior Recognition in Group Settings. in MM '23 - Proceedings of the 31st ACM International Conference on Multimedia. Proceedings of the ACM International Conference on Multimedia, no. 31st, Association for Computing Machinery, Inc, pp. 9526-9530, 31st ACM International Conference on Multimedia, Ottawa, Ontario, Canada, 29/10/23. https://doi.org/10.1145/3581783.3612858

MAGIC-TBR: Multiview Attention Fusion for Transformer-based Bodily Behavior Recognition in Group Settings. / Madan, Surbhi; Jain, Rishabh; Sharma, Gulshan et al.
MM '23 - Proceedings of the 31st ACM International Conference on Multimedia. Association for Computing Machinery, Inc, 2023. p. 9526-9530 (Proceedings of the ACM International Conference on Multimedia; No. 31st).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - MAGIC-TBR

T2 - 31st ACM International Conference on Multimedia

AU - Madan, Surbhi

AU - Jain, Rishabh

AU - Sharma, Gulshan

AU - Subramanian, Ramanathan

AU - Dhall, Abhinav

N1 - Conference code: 31st

PY - 2023/10/27

Y1 - 2023/10/27

N2 - Bodily behavioral language is an important social cue, and its automated analysis helps in enhancing the understanding of artificial intelligence systems. Furthermore, behavioral language cues are essential for active engagement in social agent-based user interactions. Despite the progress made in computer vision for tasks like head and body pose estimation, there is still a need to explore the detection of finer behaviors such as gesturing, grooming, or fumbling. This paper proposes a multiview attention fusion method named MAGIC-TBR that combines features extracted from videos and their corresponding Discrete Cosine Transform coefficients via a transformer-based approach. The experiments are conducted on the BBSI dataset and the results demonstrate the effectiveness of the proposed feature fusion with multiview attention. The code is available at: https://github.com/surbhimadan92/MAGIC-TBR

AB - Bodily behavioral language is an important social cue, and its automated analysis helps in enhancing the understanding of artificial intelligence systems. Furthermore, behavioral language cues are essential for active engagement in social agent-based user interactions. Despite the progress made in computer vision for tasks like head and body pose estimation, there is still a need to explore the detection of finer behaviors such as gesturing, grooming, or fumbling. This paper proposes a multiview attention fusion method named MAGIC-TBR that combines features extracted from videos and their corresponding Discrete Cosine Transform coefficients via a transformer-based approach. The experiments are conducted on the BBSI dataset and the results demonstrate the effectiveness of the proposed feature fusion with multiview attention. The code is available at: https://github.com/surbhimadan92/MAGIC-TBR

KW - bodily behavior

KW - dct

KW - multiview attention

KW - transformer

UR - http://www.scopus.com/inward/record.url?scp=85179553773&partnerID=8YFLogxK

UR - https://dl.acm.org/action/showFmPdf?doi=10.1145%2F3581783

U2 - 10.1145/3581783.3612858

DO - 10.1145/3581783.3612858

M3 - Conference contribution

AN - SCOPUS:85179553773

T3 - Proceedings of the ACM International Conference on Multimedia

SP - 9526

EP - 9530

BT - MM '23 - Proceedings of the 31st ACM International Conference on Multimedia

PB - Association for Computing Machinery, Inc

Y2 - 29 October 2023 through 3 November 2023

ER -

Madan S, Jain R, Sharma G, Subramanian R, Dhall A. MAGIC-TBR: Multiview Attention Fusion for Transformer-based Bodily Behavior Recognition in Group Settings. In MM '23 - Proceedings of the 31st ACM International Conference on Multimedia. Association for Computing Machinery, Inc. 2023. p. 9526-9530. (Proceedings of the ACM International Conference on Multimedia; 31st). doi: 10.1145/3581783.3612858

MAGIC-TBR: Multiview Attention Fusion for Transformer-based Bodily Behavior Recognition in Group Settings

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this