Northern Ontario School of Medicine
Purpose: As medical education evolves to competency-based programs, attending physicians will be required to make more entrustment decisions about residents’ readiness to practice unsupervised. The aim of this study was to determine if attending anesthesiologists agreed on which residents should be entrusted. We also investigated rater stringency and factors influencing raters’ entrustment decisions.
Methods: This was a mixed-methods pilot study. In phase one (quantitative), 32 attending anesthesiologists used the mini-CEX in anesthesiology to rate resident performance in nine scripted, simulation-based scenarios. Attending anesthesiologists also made entrustment decisions about the resident in each scenario. In phase two (qualitative), attending anesthesiologists were interviewed to explore factors influencing their entrustment decisions.
Results: Attending anesthesiologists had high levels of agreement for entrustment decisions in four of nine scenarios. In the other five scenarios, about half of the raters entrusted the resident and half did not entrust the resident. A pattern in rater stringency was found with anesthesiologists behaving as hawks, moderate raters, or doves. Hawks and doves comprised more than half of the raters. Raters created internal checklists, used critical behaviours (e.g. medical jargon), and formed impressions to inform their entrustment decisions. Raters’ opinion varied about which CanMEDs roles were most important and raters applied the Likert scale on the mini-CEX in an idiosyncratic fashion.
Conclusions: Attending anesthesiologists do not agree on which residents should be entrusted in simulation-based assessment. Hawks and doves are not just outlying raters and will contribute to residents being entrusted (or not entrusted) in competency-based programs.
