‘Labelling the Gaps’: A Weakly Supervised Automatic Eye Gaze Estimation

Shreya Ghosh; Abhinav Dhall; Munawar Hayat; Jarrod Knibbe

doi:10.1007/978-3-031-26316-3_44

‘Labelling the Gaps’: A Weakly Supervised Automatic Eye Gaze Estimation

Shreya Ghosh, Abhinav Dhall, Munawar Hayat, Jarrod Knibbe

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

Over the past few years, there has been an increasing interest to interpret gaze direction in an unconstrained environment with limited supervision. Owing to data curation and annotation issues, replicating gaze estimation method to other platforms, such as unconstrained outdoor or AR/VR, might lead to significant drop in performance due to insufficient availability of accurately annotated data for model training. In this paper, we explore an interesting yet challenging problem of gaze estimation method with a limited amount of labelled data. The proposed method utilize domain knowledge from the labelled subset with visual features; including identity-specific appearance, gaze trajectory consistency and motion features. Given a gaze trajectory, the method utilizes label information of only the start and the end frames of a gaze sequence. An extension of the proposed method further reduces the requirement of labelled frames to only the start frame with a minor drop in the generated label’s quality. We evaluate the proposed method on four benchmark datasets (CAVE, TabletGaze, MPII and Gaze360) as well as web-crawled YouTube videos. Our proposed method reduces the annotation effort to as low as 2.67%, with minimal impact on performance; indicating the potential of our model enabling gaze estimation ‘in-the-wild’ setup¹ (https://github.com/i-am-shreya/Labelling-the-Gaps).

Original language	English
Title of host publication	Computer Vision – ACCV 2022
Subtitle of host publication	16th Asian Conference on Computer Vision, Macao, China, December 4-8, 2022, Proceedings, Part IV
Editors	Lei Wang, Juergen Gall, Tat-Jun Chin, Imari Sato, Rama Chellappa
Place of Publication	Cham, Switzerland
Publisher	Springer Nature Switzerland AG
Pages	745-763
Number of pages	19
ISBN (Electronic)	978-3-031-26316-3
ISBN (Print)	978-3-031-26315-6
DOIs	https://doi.org/10.1007/978-3-031-26316-3_44
Publication status	Published - 2023
Externally published	Yes
Event	16th Asian Conference on Computer Vision - Macao, China Duration: 4 Dec 2022 → 8 Dec 2022

Publication series

Name	Lecture Notes in Computer Science
Volume	13844
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	16th Asian Conference on Computer Vision
Abbreviated title	ACCV 2022
Country/Territory	China
City	Macao
Period	4/12/22 → 8/12/22

Keywords

Gaze estimation
Neural network
Weakly-supervised learning

Access to Document

10.1007/978-3-031-26316-3_44Licence: In Copyright

https://rdcu.be/dLlaELicence: In Copyright

Cite this

Ghosh, S., Dhall, A., Hayat, M., & Knibbe, J. (2023). ‘Labelling the Gaps’: A Weakly Supervised Automatic Eye Gaze Estimation. In L. Wang, J. Gall, T.-J. Chin, I. Sato, & R. Chellappa (Eds.), Computer Vision – ACCV 2022: 16th Asian Conference on Computer Vision, Macao, China, December 4-8, 2022, Proceedings, Part IV (pp. 745-763). (Lecture Notes in Computer Science; Vol. 13844). Springer Nature Switzerland AG. https://doi.org/10.1007/978-3-031-26316-3_44

Ghosh, Shreya ; Dhall, Abhinav ; Hayat, Munawar et al. / ‘Labelling the Gaps’ : A Weakly Supervised Automatic Eye Gaze Estimation. Computer Vision – ACCV 2022: 16th Asian Conference on Computer Vision, Macao, China, December 4-8, 2022, Proceedings, Part IV. editor / Lei Wang ; Juergen Gall ; Tat-Jun Chin ; Imari Sato ; Rama Chellappa. Cham, Switzerland : Springer Nature Switzerland AG, 2023. pp. 745-763 (Lecture Notes in Computer Science).

@inproceedings{d987ba3bb94743eaa913bff9bb6f378b,

title = "{\textquoteleft}Labelling the Gaps{\textquoteright}: A Weakly Supervised Automatic Eye Gaze Estimation",

abstract = "Over the past few years, there has been an increasing interest to interpret gaze direction in an unconstrained environment with limited supervision. Owing to data curation and annotation issues, replicating gaze estimation method to other platforms, such as unconstrained outdoor or AR/VR, might lead to significant drop in performance due to insufficient availability of accurately annotated data for model training. In this paper, we explore an interesting yet challenging problem of gaze estimation method with a limited amount of labelled data. The proposed method utilize domain knowledge from the labelled subset with visual features; including identity-specific appearance, gaze trajectory consistency and motion features. Given a gaze trajectory, the method utilizes label information of only the start and the end frames of a gaze sequence. An extension of the proposed method further reduces the requirement of labelled frames to only the start frame with a minor drop in the generated label{\textquoteright}s quality. We evaluate the proposed method on four benchmark datasets (CAVE, TabletGaze, MPII and Gaze360) as well as web-crawled YouTube videos. Our proposed method reduces the annotation effort to as low as 2.67%, with minimal impact on performance; indicating the potential of our model enabling gaze estimation {\textquoteleft}in-the-wild{\textquoteright} setup1 (https://github.com/i-am-shreya/Labelling-the-Gaps).",

keywords = "Gaze estimation, Neural network, Weakly-supervised learning",

author = "Shreya Ghosh and Abhinav Dhall and Munawar Hayat and Jarrod Knibbe",

year = "2023",

doi = "10.1007/978-3-031-26316-3_44",

language = "English",

isbn = "978-3-031-26315-6",

series = "Lecture Notes in Computer Science",

publisher = "Springer Nature Switzerland AG",

pages = "745--763",

editor = "Lei Wang and Juergen Gall and Tat-Jun Chin and Imari Sato and Rama Chellappa",

booktitle = "Computer Vision – ACCV 2022",

address = "Switzerland",

note = "16th Asian Conference on Computer Vision, ACCV 2022 ; Conference date: 04-12-2022 Through 08-12-2022",

}

Ghosh, S, Dhall, A, Hayat, M & Knibbe, J 2023, ‘Labelling the Gaps’: A Weakly Supervised Automatic Eye Gaze Estimation. in L Wang, J Gall, T-J Chin, I Sato & R Chellappa (eds), Computer Vision – ACCV 2022: 16th Asian Conference on Computer Vision, Macao, China, December 4-8, 2022, Proceedings, Part IV. Lecture Notes in Computer Science, vol. 13844, Springer Nature Switzerland AG, Cham, Switzerland, pp. 745-763, 16th Asian Conference on Computer Vision, Macao, China, 4/12/22. https://doi.org/10.1007/978-3-031-26316-3_44

‘Labelling the Gaps’: A Weakly Supervised Automatic Eye Gaze Estimation. / Ghosh, Shreya; Dhall, Abhinav; Hayat, Munawar et al.
Computer Vision – ACCV 2022: 16th Asian Conference on Computer Vision, Macao, China, December 4-8, 2022, Proceedings, Part IV. ed. / Lei Wang; Juergen Gall; Tat-Jun Chin; Imari Sato; Rama Chellappa. Cham, Switzerland: Springer Nature Switzerland AG, 2023. p. 745-763 (Lecture Notes in Computer Science; Vol. 13844).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - ‘Labelling the Gaps’

T2 - 16th Asian Conference on Computer Vision

AU - Ghosh, Shreya

AU - Dhall, Abhinav

AU - Hayat, Munawar

AU - Knibbe, Jarrod

PY - 2023

Y1 - 2023

N2 - Over the past few years, there has been an increasing interest to interpret gaze direction in an unconstrained environment with limited supervision. Owing to data curation and annotation issues, replicating gaze estimation method to other platforms, such as unconstrained outdoor or AR/VR, might lead to significant drop in performance due to insufficient availability of accurately annotated data for model training. In this paper, we explore an interesting yet challenging problem of gaze estimation method with a limited amount of labelled data. The proposed method utilize domain knowledge from the labelled subset with visual features; including identity-specific appearance, gaze trajectory consistency and motion features. Given a gaze trajectory, the method utilizes label information of only the start and the end frames of a gaze sequence. An extension of the proposed method further reduces the requirement of labelled frames to only the start frame with a minor drop in the generated label’s quality. We evaluate the proposed method on four benchmark datasets (CAVE, TabletGaze, MPII and Gaze360) as well as web-crawled YouTube videos. Our proposed method reduces the annotation effort to as low as 2.67%, with minimal impact on performance; indicating the potential of our model enabling gaze estimation ‘in-the-wild’ setup1 (https://github.com/i-am-shreya/Labelling-the-Gaps).

AB - Over the past few years, there has been an increasing interest to interpret gaze direction in an unconstrained environment with limited supervision. Owing to data curation and annotation issues, replicating gaze estimation method to other platforms, such as unconstrained outdoor or AR/VR, might lead to significant drop in performance due to insufficient availability of accurately annotated data for model training. In this paper, we explore an interesting yet challenging problem of gaze estimation method with a limited amount of labelled data. The proposed method utilize domain knowledge from the labelled subset with visual features; including identity-specific appearance, gaze trajectory consistency and motion features. Given a gaze trajectory, the method utilizes label information of only the start and the end frames of a gaze sequence. An extension of the proposed method further reduces the requirement of labelled frames to only the start frame with a minor drop in the generated label’s quality. We evaluate the proposed method on four benchmark datasets (CAVE, TabletGaze, MPII and Gaze360) as well as web-crawled YouTube videos. Our proposed method reduces the annotation effort to as low as 2.67%, with minimal impact on performance; indicating the potential of our model enabling gaze estimation ‘in-the-wild’ setup1 (https://github.com/i-am-shreya/Labelling-the-Gaps).

KW - Gaze estimation

KW - Neural network

KW - Weakly-supervised learning

UR - http://www.scopus.com/inward/record.url?scp=85151066033&partnerID=8YFLogxK

U2 - 10.1007/978-3-031-26316-3_44

DO - 10.1007/978-3-031-26316-3_44

M3 - Conference contribution

AN - SCOPUS:85151066033

SN - 978-3-031-26315-6

T3 - Lecture Notes in Computer Science

SP - 745

EP - 763

BT - Computer Vision – ACCV 2022

A2 - Wang, Lei

A2 - Gall, Juergen

A2 - Chin, Tat-Jun

A2 - Sato, Imari

A2 - Chellappa, Rama

PB - Springer Nature Switzerland AG

CY - Cham, Switzerland

Y2 - 4 December 2022 through 8 December 2022

ER -

Ghosh S, Dhall A, Hayat M, Knibbe J. ‘Labelling the Gaps’: A Weakly Supervised Automatic Eye Gaze Estimation. In Wang L, Gall J, Chin TJ, Sato I, Chellappa R, editors, Computer Vision – ACCV 2022: 16th Asian Conference on Computer Vision, Macao, China, December 4-8, 2022, Proceedings, Part IV. Cham, Switzerland: Springer Nature Switzerland AG. 2023. p. 745-763. (Lecture Notes in Computer Science). doi: 10.1007/978-3-031-26316-3_44