Is our assessment of resuscitation skills robust? Examining the validation process

Amy Seymour-Walsh; Paul Worley; Anna Vnuk; Hugh Grantham

doi:10.1016/j.resuscitation.2016.07.144

Abstract

Purpose: Just as a chain is only as strong as its weakest link, so too is assessment only as robust as the tool used to examine performance. We sought to assess paramedic students insert a laryngeal mask airway (LMA) and intraosseous (IO) device in a simulated setting in a randomised controlled trial. We searched the literature for validated assessment tools for these two skills, and after finding none suitable to this context, we developed one.

Methods: We invited expert pre-hospital clinicians (retrievalists and intensive care paramedics) to participate in a modified Delphi study1 to investigate expert consensus for LMA and IO application in the pre-hospital setting. Two rounds of data collection were performed.

Results: Using this procedure as reported in the literature, consensus was achieved for IO (Cronbach's α = 0.725) but not for LMA (Cronbach's α = 0.494).

Conclusions: A critique of the Delphi method challenges (1) Cronbach's α as the accepted coefficient2 in reporting data agreement, (2) the current understanding of “consensus”, (3) whether a Delphi method is the best approach to determine consensus for all types of skills, and (4) the task of validation as a single, finite activity.3

Original language	English
Pages (from-to)	e60-e60
Number of pages	1
Journal	Resuscitation
Volume	106
Issue number	Supplement 1
DOIs	https://doi.org/10.1016/j.resuscitation.2016.07.144
Publication status	Published - Sept 2016
Event	European Resuscitation Congress 2016 - Reykjavik, Iceland Duration: 24 Sept 2016 → 25 Sept 2016

Keywords

Assessment for learning
Performance assessment
Resuscitation Skills
Laryngeal mask airway
Intraosseous
randomised controlled trial
Delphi method

Access to Document

10.1016/j.resuscitation.2016.07.144Licence: In Copyright

Cite this

@article{1b86af198a12430c87dbe979811b0beb,

title = "Is our assessment of resuscitation skills robust? Examining the validation process",

abstract = "Purpose: Just as a chain is only as strong as its weakest link, so too is assessment only as robust as the tool used to examine performance. We sought to assess paramedic students insert a laryngeal mask airway (LMA) and intraosseous (IO) device in a simulated setting in a randomised controlled trial. We searched the literature for validated assessment tools for these two skills, and after finding none suitable to this context, we developed one.Methods: We invited expert pre-hospital clinicians (retrievalists and intensive care paramedics) to participate in a modified Delphi study1 to investigate expert consensus for LMA and IO application in the pre-hospital setting. Two rounds of data collection were performed.Results: Using this procedure as reported in the literature, consensus was achieved for IO (Cronbach's α = 0.725) but not for LMA (Cronbach's α = 0.494).Conclusions: A critique of the Delphi method challenges (1) Cronbach's α as the accepted coefficient2 in reporting data agreement, (2) the current understanding of “consensus”, (3) whether a Delphi method is the best approach to determine consensus for all types of skills, and (4) the task of validation as a single, finite activity.3",

keywords = "Assessment for learning, Performance assessment, Resuscitation Skills, Laryngeal mask airway, Intraosseous, randomised controlled trial, Delphi method",

author = "Amy Seymour-Walsh and Paul Worley and Anna Vnuk and Hugh Grantham",

year = "2016",

month = sep,

doi = "10.1016/j.resuscitation.2016.07.144",

language = "English",

volume = "106",

pages = "e60--e60",

journal = "Resuscitation",

issn = "0300-9572",

publisher = "Elsevier Ireland Ltd",

number = "Supplement 1",

note = "European Resuscitation Congress 2016, ERC 2016 ; Conference date: 24-09-2016 Through 25-09-2016",

}

TY - JOUR

T1 - Is our assessment of resuscitation skills robust? Examining the validation process

AU - Seymour-Walsh, Amy

AU - Worley, Paul

AU - Vnuk, Anna

AU - Grantham, Hugh

PY - 2016/9

Y1 - 2016/9

N2 - Purpose: Just as a chain is only as strong as its weakest link, so too is assessment only as robust as the tool used to examine performance. We sought to assess paramedic students insert a laryngeal mask airway (LMA) and intraosseous (IO) device in a simulated setting in a randomised controlled trial. We searched the literature for validated assessment tools for these two skills, and after finding none suitable to this context, we developed one.Methods: We invited expert pre-hospital clinicians (retrievalists and intensive care paramedics) to participate in a modified Delphi study1 to investigate expert consensus for LMA and IO application in the pre-hospital setting. Two rounds of data collection were performed.Results: Using this procedure as reported in the literature, consensus was achieved for IO (Cronbach's α = 0.725) but not for LMA (Cronbach's α = 0.494).Conclusions: A critique of the Delphi method challenges (1) Cronbach's α as the accepted coefficient2 in reporting data agreement, (2) the current understanding of “consensus”, (3) whether a Delphi method is the best approach to determine consensus for all types of skills, and (4) the task of validation as a single, finite activity.3

AB - Purpose: Just as a chain is only as strong as its weakest link, so too is assessment only as robust as the tool used to examine performance. We sought to assess paramedic students insert a laryngeal mask airway (LMA) and intraosseous (IO) device in a simulated setting in a randomised controlled trial. We searched the literature for validated assessment tools for these two skills, and after finding none suitable to this context, we developed one.Methods: We invited expert pre-hospital clinicians (retrievalists and intensive care paramedics) to participate in a modified Delphi study1 to investigate expert consensus for LMA and IO application in the pre-hospital setting. Two rounds of data collection were performed.Results: Using this procedure as reported in the literature, consensus was achieved for IO (Cronbach's α = 0.725) but not for LMA (Cronbach's α = 0.494).Conclusions: A critique of the Delphi method challenges (1) Cronbach's α as the accepted coefficient2 in reporting data agreement, (2) the current understanding of “consensus”, (3) whether a Delphi method is the best approach to determine consensus for all types of skills, and (4) the task of validation as a single, finite activity.3

KW - Assessment for learning

KW - Performance assessment

KW - Resuscitation Skills

KW - Laryngeal mask airway

KW - Intraosseous

KW - randomised controlled trial

KW - Delphi method

U2 - 10.1016/j.resuscitation.2016.07.144

DO - 10.1016/j.resuscitation.2016.07.144

M3 - Meeting Abstract

SN - 0300-9572

VL - 106

SP - e60-e60

JO - Resuscitation

JF - Resuscitation

IS - Supplement 1

T2 - European Resuscitation Congress 2016

Y2 - 24 September 2016 through 25 September 2016

ER -

Is our assessment of resuscitation skills robust? Examining the validation process

Abstract

Keywords

Access to Document

Fingerprint

Cite this