Friday, August 7, 2015

Validity Evidence for a Newly-Developed Simulator-Based Assessment to Evaluate Error Identification and Recovery Skills in Surgical Residents

Shari Meyerson, MD, MEd
Northwestern Memorial Hospital
Background: Although error identification and recovery skills are essential for the safe practice of surgery they have not traditionally been taught or evaluated in residency training.

Aims: This study aims to develop and validate a method for assessing error identification and recovery skills in surgical residents using a thoracoscopic lobectomy simulator.

Methods: A five station, simulator-based exam was developed containing the most commonly encountered cognitive and technical errors occurring during division of the superior pulmonary vein for left upper lobectomy. Successful completion of each station requires identification and correction of these errors. Exams were video-recorded and scored in a blinded fashion using an exam-specific rating instrument evaluating task performance as well as error identification and recovery skills. Evidence of validity was collected in the categories of content, response process, internal structure and relationship to other variables.

Results: Fifteen general surgical residents (9 interns, 6 third year residents) completed the exam. Content validity was ensured by basing the exam on prior work demonstrating the most common significant errors created by residents in this simulated procedure. Inter-rater reliability was high with intraclass correlation coefficient of 0.78 between four trained raters. Station difficulty ranged from 0.64-0.84. All stations adequately discriminated between high and low performing residents with discrimination ranging from 0.35-0.65. The overall exam score was significantly higher for intermediate residents than interns (mean score 74 vs. 64, p=0.03).

Conclusions: The described simulator-based exam with embedded errors and its accompanying assessment tool can be used to measure error identification and recovery skills in surgical residents. This exam provides a valid method for comparing teaching strategies designed to improve error prevention, recognition and recovery in residents to improve patient safety.