Utility of Item Response Theory in medical students’ general procedural skill assessment

Hirotaka Kato
University of Kentucky / Albert B. Chandler

Background: Though Item Response Theory (IRT) is a well-established method to analyze response data, the utility of IRT in clinical skill assessment is not well-described. Using the procedure simulation test data whose standards were determined by the Angoff method, we explored (1) the psychometric properties of the test and (2) how different passing scores change the interpretations of the test.

Methods: Our study was conducted at a single US medical school where 3rd-year medical students are required to pass 10 procedural skill simulations. The unidimensional partial credit model was applied to the data of 131 students whose total scores for each procedure were recorded. We mapped person locations and thresholds on the same continuum (i.e., the item-person map) which allowed us to assess how the test performance would change by different passing scores (i.e., Angoff, +1, +2, and perfect scores).

Results: The item-person map elucidated a gap between student abilities and item difficulties (thresholds). The Angoff scores were too easy to differentiate high- from low-performers. Adding 1 point to the Angoff scores resulted in the largest variance, that would allow the test to assess a wider range of skills. This procedural skill test however was not suited to assess a wide range of skills unless we make some items more difficult and appropriate for the target populations.

Conclusions:  This study illustrates the application of IRT in clinical skill assessment. The results showed that IRT was useful to refine the test itself and the passing scores.

Employing Strategies to Address Implicit Racial Bias in the Home Visit Setting: A Phenomenological Study of Family Medicine Residents

Anna Goroncy 

The Christ Hospital / University of Cincinnati 

Background: Implicit racial bias (IB) in physicians contributes to racial health inequities. Residents are not consistently trained to address IB. Few curricula addressing IB in graduate medical education have been evaluated, especially in the clinical setting. 

Objectives: The purpose of this phenomenological study is to characterize Family Medicine (FM) residents’ experience of employing strategies to mitigate IB during primary care home visits (HVs) to urban, predominately African-American, homebound older adults. The outcomes of this qualitative study will inform future curriculum development. 

Methods: FM residents completed pre-work, including taking the Implicit Association Test and evaluating strategies to address IB. Residents applied these strategies during HVs to homebound older adults. Residents completed written reflections about their experiences and commitments-to-change (CTC). A survey two months later assessed completion of targeted actions and barriers faced. Resident focus groups were utilized to enhance themes drawn from reflections. Researchers completed a thematic analysis of this data January-July 2020. 

Results: Thematic analysis identified five themes: Response to IAT, barriers, strategies, value of HVs and mindfulness definition. In follow-up surveys, all residents’ stated level of CTC remained the same (9/9, 100%) and 8/9 residents (89%) had partially or fully implemented their intended change at 2 months. 

Conclusions: Residents utilized the opportunity to learn and apply strategies to address IB. Residents continued to implement newly-learned strategies in the clinical setting two months after training and applied skills to settings outside of HVs and other bias types. These findings can facilitate development of meaningful, clinically-based IB curricula with lasting impacts.

Utilizing High-Fidelity Simulation as an Approach for Senior Pediatric Residents to Further Develop Their Teaching Skills and to Enhance Their Delivery of Feedback to Junior Learners

Kheyandra Lewis

Drexel University / St. Christopher's Hospital for Children

Background: The Accreditation Council for Graduate Medical Education mandates that residents must receive specific instruction on how to teach others during their training. Many residency programs fulfill this requirement through various resident-as-teacher (RaT) curricula. At St. Christopher’s Hospital for Children, the pediatric residency program employs a high-fidelity simulation that is created, facilitated, and debriefed by a third-year pediatric resident. The goal of the activity is to provide the third-year resident with the opportunity to teach in a venue in which they have minimal experience.

Aim: The main aim of this study was to assess the perspectives of third-year pediatric residents who conducted a RaT simulation as well the perspectives of first- and second-year pediatric residents who participated, in regard to comfort with teaching and delivering feedback using simulation.

Methods: A convergent mixed-methods study approach was used to assess resident perspectives on teaching and feedback delivery through the administration of surveys and semi-structured interviews.

Results: 37% of third-year residents felt “very comfortable” using simulation to teach junior learners and 62.5% felt “very comfortable” providing feedback to junior learners using simulation after completion of the RaT simulation.

Conclusions: The inclusion of a resident created, designed, and debriefed simulation as part of a RaT curriculum can help to improve overall comfort in teaching and feedback delivery, and may in turn impart valuable skills that can be applied to other educational sessions.

Structured Formative Feedback After Physical Exam Skills Session

Christopher Sumski
Children's Hospital of Wisconsin

Collegiality amongst emergency physicians: what is it and can it be defined?

James Mallows
Nepean Hospital

The Quality Improvement Pre-Training Initiative for Medical Students During the Surgical Clerkship: How can Quality Improvement Learning and Project Planning Be Accomplished in Four Weeks?

Alexandra Hollingworth
HonorHealth John C. Lincoln Medical Center

Exploring Professional Identity Formation and Communities of Practice Through Anesthesiology Resident Confessions: A Retrospective Qualitative Study

Matthew Hirschfeld
University of Rochester


Background: “Confessions” is a program leadership-moderated explicit learning activity in which residents discuss anonymously submitted confessions. Confessions allows a safe space for resident feedback, permits an opening for discussions on Professional Identity Formation (PIF), and the opportunity for identification with a Community of Practice (CoP).

Objective: This retrospective qualitative study analyzed typed confessions from Clinical Base Year residents to identify themes impacting resident PIF and participation within a CoP. We also investigated confessions from senior anesthesiology residents and from piloted interdisciplinary Confessions for progression of these themes throughout residency training.

Methods: We analyzed 244 confessions collected during 21 Confessions conducted between October, 2016 and anuary, 2020 at a single institution.

In keeping with Grounded Theory and the Constant Comparative Method, confessions were qualitatively coded in three stages: open, axial, and selective. Researcher triangulation, a code book, and member checking were used to enhance the study’s trustworthiness. The concepts of PIF and CoP served as the theoretical perspective for the study.

Results: Interrelated, longitudinal themes associated with PIF and CoP, such as “support” and “knowledge,” were uncovered. Longitudinal themes were related to important transitions during residency training and evolved over time. Episodic themes were related to local concerns.

Conclusions: Transition periods are important milestones in physician PIF and identification within a CoP. The graduate medical community should continue to focus efforts to allow residents time for studying and self- care during normal clinical hours, as well as on teaming with non- physician healthcare workers. Confessions is an easily adaptable activity for other programs and institutions.