TY - JOUR
T1 - Food for thought
T2 - Commentary on Burnette et al. (2021) “Concerns and recommendations for using Amazon MTurk for eating disorder research”
AU - Moeck, Ella K.
AU - Bridgland, Victoria M.E.
AU - Takarangi, Melanie K.T.
PY - 2022/2
Y1 - 2022/2
N2 - Burnette et al. aimed to validate two eating disorder symptom measures among transgender adults recruited from Mechanical Turk (MTurk). After identifying several data quality issues, Burnette et al. abandoned this aim and instead documented the issues they faced (e.g., demographic misrepresentation, repeat submissions, inconsistent responses across similar questions, failed attention checks). Consequently, Burnette et al. raised concerns about the use of MTurk for psychological research, particularly in an eating disorder context. However, we believe these claims are overstated because they arise from a single study not designed to test MTurk data quality. Further, despite claiming to go “above and beyond” current recommendations, Burnette et al. missed key screening procedures. In particular, they missed procedures known to prevent participants who use commercial data centers (i.e., server farms) to hide their true IP address and complete multiple surveys for financial gain. In this commentary, we outline key screening procedures that allow researchers to obtain quality MTurk data. We also highlight the importance of balancing efforts to increase data quality with efforts to maintain sample diversity. With appropriate screening procedures, which should be preregistered, MTurk remains a viable participant source that requires further validation in an eating disorder context.
AB - Burnette et al. aimed to validate two eating disorder symptom measures among transgender adults recruited from Mechanical Turk (MTurk). After identifying several data quality issues, Burnette et al. abandoned this aim and instead documented the issues they faced (e.g., demographic misrepresentation, repeat submissions, inconsistent responses across similar questions, failed attention checks). Consequently, Burnette et al. raised concerns about the use of MTurk for psychological research, particularly in an eating disorder context. However, we believe these claims are overstated because they arise from a single study not designed to test MTurk data quality. Further, despite claiming to go “above and beyond” current recommendations, Burnette et al. missed key screening procedures. In particular, they missed procedures known to prevent participants who use commercial data centers (i.e., server farms) to hide their true IP address and complete multiple surveys for financial gain. In this commentary, we outline key screening procedures that allow researchers to obtain quality MTurk data. We also highlight the importance of balancing efforts to increase data quality with efforts to maintain sample diversity. With appropriate screening procedures, which should be preregistered, MTurk remains a viable participant source that requires further validation in an eating disorder context.
KW - data quality
KW - Mechanical Turk
KW - online recruitment platform
KW - online research
KW - open science
KW - survey research
UR - http://www.scopus.com/inward/record.url?scp=85122284574&partnerID=8YFLogxK
U2 - 10.1002/eat.23671
DO - 10.1002/eat.23671
M3 - Comment/debate
C2 - 34984704
AN - SCOPUS:85122284574
SN - 0276-3478
VL - 55
SP - 282
EP - 284
JO - International Journal of Eating Disorders
JF - International Journal of Eating Disorders
IS - 2
ER -