Karen Mangold, MD, MEd
Ann & Robert H. Lurie Children's Hospital
Background: Residency programs are developing new assessment tools to look at resident competence. New methods, such as simulation, are being used to evaluate resident competence. Assessment tools used during simulations need to be reliable and valid if they are to be used towards summative evaluations.
Objective/Purpose: This study is to evaluate the validity of new assessment tools that will be used to assess residents’ clinical and team skills during simulated resuscitations.
Methods: Three checklists were developed and applied to cases during an assessment program. Cases were video-recorded and reviewed by raters. Results were examined for reliability, including a generalizability analysis.
Results: The checklists show good inter-rater reliability, with intra-class coefficients above 0.75 for all three tools. A generalizability study demonstrates variance is due to the affects of participants, question items and the interaction between these two facets. A design study demonstrates good reliability using the current number of checklist items. One rater yielded good reliability on the apnea and seizure stations, but the ventricular tachycardia station requires two raters to achieve acceptable reliability.
Conclusion: These three checklist tools are reasonably reliable and valid, and may be used for assessment of pediatric resident resuscitation skills.
