Understanding expertise in performance-based assessment

Introduction: Much of the assessment literature in health professions education has focused on ensuring that assessments are valid and reliable. It has traditionally been assumed that the assessor poses the main threat to validity and reliability and efforts have, therefore, focused on approaches to mitigate assessor influence. More recently, this position has been challenged by the alternative view that the assessor should have an active role in enhancing assessment validity. The nature of this active role and expertise in informing assessment judgements and decision-making, however, is not yet sufficiently understood.
Methods: This study sought to understand assessment expertise and its role in performance-based assessments such as the objective structured clinical examination by conducting a hermeneutic literature review. A systematic search strategy was developed to identify articles relevant to assessor decision-making during performance based assessments of medical doctors. Title and abstract screening of the initial search results (n = 1495) was conducted by two authors (SK and AS-W). Twenty-four articles were identified and underwent full text screening. Eight articles were excluded as they did not strictly meet the eligibility criteria. The resulting 16 articles were analysed against pre-determined questions relating to assessment expertise, decision-making, and training. As is consistent with a hermeneutic literature review, we expanded our dataset through purposive sampling until data saturation was reached and no new questions were generated. The findings were analysed thematically using situativity and complexity theory.