Impact of an Asynchronous Trainee-as-Teacher Curriculum on Teaching Self-Efficacy and Performance

Kathleen Timme
University of Utah

Introduction

Physicians are responsible for educating their patients, peers, and health professional learners. Therefore, it is crucial for trainees to have opportunities to develop and hone teaching skills during training. Resident-as-teacher curricula have developed in response to this need, but are largely classroom-based with limited opportunities to practice these valuable teaching skills. In the midst of the COVID-19 pandemic and clinical demands that challenge the feasibility of live classroom-based learning experiences, there is a need for online asynchronous learning opportunities for trainees. The aim of this study was to investigate the impact of an asynchronous online trainee-as-teacher curriculum on teaching performance and self-efficacy. 

Methods

Resident physicians enrolled in an elective program featuring four online modules, real-life teaching experiences, and a reflective writing assignment. Participants completed a self-efficacy assessment before and after completion of the curriculum. They also collected feedback forms from the individuals they taught during the teaching experiences. We applied the qualitative methodology grounded theory in analyzing the written reflection. 

Results

Self-efficacy improved after program completion, with 100% of participants indicating that they felt comfortable teaching patients, colleagues/attendings at or above their level of training, and learners junior to themselves. Teaching evaluations from patients and small group learners were also favorable. Key themes from the qualitative portion of the study support that participants gained knowledge about health literacy, small group teaching, and clinical teaching. 

Discussion

This pilot study showed that an online asynchronous trainee-as-teacher program with opportunities for teaching practice and self-reflection fostered development of self-efficacy and improved teaching performance.

The Residency Interview Prep Program: A novel curriculum to prepare medical students for residency interview

Ryann Quigley
New York University School of Medicine

Background: Applying and matching into residency programs is one of the most stressful and high stakes processes a medical student can go through during their training. A large variable in a student’s likelihood of matching rests in the interview performance. Given the importance of the residency interview shown in the 2020 Program Director Survey from the National Residency Matching Program, NYU Grossman School of Medicine expanded upon existing curriculum to develop a novel curriculum to better prepare students for the residency interview and use faculty time efficiently, known as the Residency Interview Coaching Sessions.

Aims: To increase student’s knowledge about the interview process, skill practice, feedback, reflection, and simulates the authentic residency program interview.

Methods: A mixed-methods study was conducted through a post-event survey examining student self-reported confidence and preparedness using a retrospective pre-post Likert-score survey. 62 of 88 students invited participated in the Residency Interview Coaching Program.

Results: 34 students submitted an evaluation, and those participated in the coaching sessions rated the sessions highly (m=4.64 of 5.00). Additionally, the differences in mean scores between pre- and post-surveys demonstrated showed statistical significance indicating increases in students reported ratings of confidence and preparedness for the interview process. Content analysis was conducted on qualitative answers that demonstrate this curriculum was valuable to students.

Conclusions: Performance during the residency interviews is the most important factor in ranking applicants. The Residency Interview Coaching Sessions helped students feel more prepared and confident prior to interviews.

Improving Clinical Feedback after Transitions of Care

Tammy Ruth
Seattle Children's Hospital

Background: Transitions of patient care often occur before the final diagnosis or outcome is known; physicians follow-up when motivated by curiosity or personal connection and when time allows, without a widely used platform to facilitate learning or to capture patients who experience clinical worsening.

Aims: This quantitative study describes the impact of an automated provider email sent after a patient returns to the ED within 72 hours and is subsequently admitted to an inpatient service. We determined the usefulness of this intervention by identifying (1) the actions taken by the recipient of an email, (2) the rationale for these actions, and (3) their preferences for receiving this type of clinical feedback. 

Methods: We conducted an online cross-sectional survey of a convenience sample comprised of 131 pediatric residents, pediatric emergency medicine fellows and attendings at a large, single-site, west coast academic children’s hospital. 

Results: Overall response rate was 47%. Residents (100%), fellows (100%), and attendings (95.65%) read the email, with 45.65% indicating the email prompted them to do something different from how they typically follow-up on ED patients. Respondents spent additional time reflecting (84.78%), reviewed the patient’s electronic medical record (73.91%), read more about the patient’s condition (30.43%), discussed the patient’s case with someone else (26.09%), and made changes to their practice (21.74%). For this type of clinical feedback, 82.22% preferred to continue to the current notification system.

Conclusions: Residents, fellows, and attendings find this system educationally helpful for clinical feedback after patient transitions and report changes to their medical practice following the receipt of an automated email. 

Qualitative Analysis of Feedback in Primary Care Resident Clinic

Shelley Ost

UT Methodist Physicians

Background: Residents spend a great deal of time with supervising faculty in their continuity clinic in primary care residencies. The long-term relationship between the resident and faculty presents an opportunity for providing feedback on the resident’s knowledge, skills, and practice style. 

Methods: Focus group discussions with primary care residency supervising faculty at the University of Tennessee Health Science Center were analyzed to determine the goals, current knowledge and feelings, and barriers regarding providing feedback to residents during continuity clinic. Qualitative analysis using grounded theory was performed to identify themes within the focus group data, leading to an underlying theory about the provision of feedback to residents in primary care resident clinic. Data were used to propose improvements to the current process of feedback.

Results: Categories in the data included the longitudinal nature of the faculty-resident relationship, need for management of emotional response to feedback, faculty preparedness, and difficulty determining the content of feedback that should be provided. These influenced the three major components of feedback, which were the content, delivery, and reception of feedback.

Conclusion: Addressing components of the content of feedback, delivery of feedback, and the attitude or receptivity of the learner can help improve the quality and quantity of feedback given to residents in clinic.

Simulation Based Mastery Learning Improves the Performance of Donning and Doffing of Personal Protective Equipment by Medical Students

Danielle Miller

Stanford University School of Medicine

Introduction: Medical students lack adequate training on how to correctly don and doff personal protective equipment (PPE). Simulation-based mastery learning (SBML) is an effective technique for procedural education. The aim of this study is to determine if SBML improves proper PPE donning and doffing by medical students.

 

Methods: This was a prospective, pretest-post-test study of 155 medical students on demonstration of correct PPE use before and after a SBML intervention.  Subjects completed standard hospital training by viewing a CDC training video on proper PPE use prior to the intervention. They then participated in a SBML training session that included baseline testing, deliberate practice with expert feedback, and post-testing until mastery was achieved. Students were assessed using a previously developed 21-item checklist on donning and doffing PPE with a minimum passing standard (MPS) of 21/21 items. Differences between pretest and post-test scores were analyzed using paired t-tests. Students at preclinical and clinical levels of training were compared with an independent t-test. 

Results: Two participants (1.3%) met the MPS on pretest. Of the remaining 153 subjects who participated in the intervention, 151 (98.7%) reached mastery. Comparison of mean scores from pretest to final post-test significantly improved from an average raw score of 12.55/21 (standard deviation [SD] = 2.86), to 21/21(SD = 0), t(150) =36.3, p<0.001. There was no difference between pretest scores of preclinical and clinical students.  

Conclusion: SBML improves medical student performance in PPE donning and doffing in a simulated environment. This approach standardizes PPE training for students in advance of clinical experiences.

Active Learning in CME: Creation of an objective observation tool

Yemisi Jones
Cincinnati Children's Hospital


Introduction: Despite the evidence supporting the use of active learning strategies to enhance learning, they are infrequently used during continuing education (CE) lectures, in both in-person and virtual settings. Studies of active learning during didactic sessions have predominantly involved novice learners who require more intensive active learning strategies. This project aims to create an observation tool for use in program evaluation.

Methods: We reviewed the literature for active learning techniques described in large-group medical education. These strategies were then rated by 2 separate Delphi panels, one made of medical education experts (n = 9) and one of CE learners (n = 12). Each panel completed two rounds of ratings in an effort to achieve consensus. Experts were asked to rate each strategy for appropriateness and feasibility for both in-person and virtual session use. Attendees rated each strategy for likelihood to engage in an in-person and virtual format. The results of the Delphi panel informed which strategies were included in the Observed Active Learning Tool (OALT). Further validity evidence for the tool was collected by assessing inter-rater reliability using Fleiss’ kappa of four raters (2 clinicians and 2 non-clinicians). Iterative adjustments were made to the tool to achieve optimal reliability.

Results: The panels rated 31 active learning strategies and found the majority (25) to be appropriate and feasible for in-person (27) and virtual (25) CE large-groups. The attendee panel reached consensus on high likelihood to engage with 18 strategies in both in-person and virtual settings. There were 20 strategies which reached positive consensus with both groups and none which were rated poorly by both panels. Initial inter-rater reliability of the tool was moderate (kappa .53-.63). After modifications to the tool, final kappa was ***.

Discussion: Experts and CE attendees rated highly most of the active learning strategies from the literature for both virtual and in-person learning environments. The list of strategies rated highly by both panels can serve as a guide for CE planners and educators seeking to incorporate active learning into large-group learning. In addition the OALT may be used by both clinical and non-clinical CE professionals to guide program evaluation and improvement.

Perceptions of Feedback Delivery by Surgical Educators in the Era of Gender Diversity Awareness

Michelle Clarke
Mayo Clinic


Background: The primary goal of authentic feedback is to improve a learner’s future performance by providing accurate, actionable, and constructive critiques of performance. Unfortunately, quality of feedback is known to be influenced by gender. Recent societal and institutional commitments to diversity, inclusion, and equity may also influence the feedback process. This study explored the perceptions of experienced anesthesia and surgical educators of providing authentic feedback, including the impact of gender.

Methods: Eleven semi-structured interviews were conducted with faculty in anesthesia and surgical subspecialties. Open-ended questions probed the climate of feedback, the participant’s current feedback processes, barriers to authentic feedback, and the perceived ability to provide authentic feedback to minority trainees. The Braun and Clarke method of thematic analysis was employed, and a narrative composed proceeding from description to interpretation to explanation.

Results: The major barrier to feedback identified was the trainee’s real or anticipated negative reactions to corrective feedback, which was independent of gender. Participants perceived a reduction in authentic feedback due to fear of trainee reprisals. Individuals in departments with greater gender parity or who had higher formal education roles were more facile and confident in providing corrective feedback. Participants felt that educating faculty on feedback delivery and trainees on feedback reception would be valuable.

Conclusions: Faculty perceived that the biggest barrier to corrective feedback was the anticipated or real reaction of the trainee. While gender likely plays a role in feedback delivery, this was not the primary driver of inauthentic feedback.

Going viral: a scoping review of the current state and impact of online research dissemination in emergency medicine

James Gray

Cincinnati Children's Hospital

The use of free, open-access medical education (FOAM) and other online methods for the dissemination of new knowledge and research continues to increase. This scoping review aims to map and synthesize the literature describing the use of novel, online tools for the dissemination of emergency medicine research and create a conceptual model representing a modern understanding of dissemination for researchers. We searched the traditional literature via Pubmed, CINAHL, EMBASE, ERIC, and SCOPUS, and the grey literature via internet search engines. We included 73 out of 874 abstracts and articles identified, 58 of which were from the traditional literature. We identified 7 primary domains represented in the literature: integration with traditional sources, quantitative measurement of novel dissemination methods, online communities of practice, professional development, quality assessment, advantages and disadvantages of FOAM, and a scoping review of FOAM for knowledge translation. Online methods to disseminate new research are reaching new importance for researchers, but significant gaps exist in the understanding and measurement of its potential. Researchers and research networks should leverage their skill and knowledge to help provide better understanding of these novel tools. 

Addressing the Pediatric Rheumatology Workforce Shortage Through a Dedicated Early Exposure Elective in Pediatric Rheumatology

Katherine Schultz

Cincinnati Children's Hospital

Objective

The pediatric rheumatology workforce shortage is worsening as few medical graduates pursue this subspecialty. The root cause remains unclear, but research suggests the lack of early exposure may contribute. We created DEEEPR (Dedicated Early Exposure Elective in Pediatric Rheumatology) and assessed the impact of early exposure on medical student interest in pediatric rheumatology. We hypothesized DEEEPR would generate interest in pursuing pediatric rheumatology. 

Methods

DEEEPR, designed via adult learning-theories, was introduced in 2016 and included a weeklong experience for third-year medical students and 2- or 4-week long experiences for fourth-year medical students. A prospective, survey-based educational study was conducted in 2018-2019 to evaluate the impact of DEEEPR on career discernment. 

Results

In the four years following the implementation of DEEEPR, the number of third-year medical students rotating on pediatric rheumatology ranged from 24-43 students. This was a notable increase from 8 students the year prior to DEEEPR implementation. The number of fourth-year medical students rotating in pediatric rheumatology did not change as a result of DEEEPR. 

Following elective participation, 38/59 (60%) of all students indicated they had increased interest in pursuing pediatric rheumatology. On a follow up survey assessing long-term impact of the curriculum, 13/34 (38%) participated, and none of the 13 indicated they still wanted to pursue pediatric rheumatology. 

Conclusion: 

DEEEPR was successful in increasing the number of third-year medical students exposed to pediatric rheumatology and increased initial interest in a career in pediatric rheumatology, although this interest was not maintained long-term. 

CHANGE-Asthma: A Mixed Methods Evaluation of a Novel Educational App for Asthma

David Swetland

Cincinnati Children's Hospital

Background: Asthma education is an important component of asthma management and control. With the COVID-19 pandemic, standard asthma education was affected by a transition to telehealth visits, preventing hands-on education experiences for pediatric patients and their caregivers.

Aims: The goal of this study was to evaluate the effects and utility of the CHANGE-Asthma app, an educational bundle of augmented reality-based smart games, educational videos, and an asthma action plan, on caregivers of pediatric patients with asthma who have been seen in subspecialty telehealth clinics.

Methods: This was an explanatory, sequential mixed methods study. Quantitative data consisted of a childhood Asthma Control Test (cACT), a knowledge assessment, and time spent within the app. Qualitative data consisted of semi-structured interviews and evaluation by thematic analysis to develop recurring themes related to their experience. 

Results: There was no statistically significant change in the cACT or knowledge assessment scores when evaluating pre- and post-intervention scores. Time spent within the app significantly decreased after the first week; the mean total time spent within the app was 8.6 minutes over an average of 14 weeks (range 9-19 weeks). Several themes regarding the app developed from thematic analysis: usability, real-world application, who might benefit the most, saturation of knowledge, response to augmented reality, and novel ideas for future iterations. Caregivers stated how they utilized augmented reality as deliberate practice to improve their skills. 

Conclusion: With this mixed methods approach, we evaluated caregivers’ real-world usage of a mobile health app and obtained personal insight into the utility of novel technologies for patient education. 

Validation and use of an Induction of General Endotracheal Anesthesia Checklist Assessment for Medicals Students in a Clinical Setting during their Introductory Anesthesiology Clerkship

Wendy Nguyen
University of Minnesota Medical School

Background:

The American Association of Medical Colleges deemed performing lifesaving procedures such as airway management a medical student competency for transitioning to residency. Anesthesiology clerkships provide the unique opportunity for medical students to practice these procedures in the safe and controlled operating room environment. Given limitation of practicing procedures in simulation centers, we aim to validate a Checklist we developed for use in the clinical setting where medical students perform the steps of safely inducing general anesthesia with endotracheal intubation. 

Methods:

We validated the Checklist by aligning the Checklist with our clerkship objectives, modified it based on feedback and after trialing it. Using paired t-tests, we calculated pre- and post-clerkship Checklist scores to determine curriculum efficacy. The baseline Checklist was done at a simulation center, whereas the post-clerkship Checklist was performed in the clinical setting. The P value of <0.05 was determined to be statistically significant. We examined agreement between overall scores with intraclass correlation coefficients (ICC).

Results:

Fourteen medical students participated in the study.  The ICC for agreement was 0.881 (95% CI: 0.357, 0.969) and ICC for consistency was 0.934 (95% CI: 0.807, 0.978). There was a statistically significant improvement in the score from baseline to final evaluation of 3.9 points (95% CI: 2.5, 5.2; p<0.001).

Conclusions:

We validated our Checklist. The statically significant change in Checklist scores suggests that our medical students gained knowledge and experience performing procedures during the clerkship and were able to demonstrate it during the Checklist assessment in a clinical environment.  The Checklist may be used as an objective measure for medical students to achieve competency.

A Collaborative Qualitative Research Study to Develop a Content-Valid Rubric for Distance Vision Subjective Manifest Refractometry

Amy Jost
Cincinnati Eye Institute

Background: Refractometry a skill that is difficult for beginners to learn and challenging for the evaluators to assess. A well-constructed rubric can provide guidance as a training tool to the novice refractionist and as an assessment tool to the instructor. 

Purpose: A new rubric was created to provide clinical educators with a content-valid, internationally-standardized rubric to teach and assess competency in subjective refractometry. 

Methods: A panel of seven international content experts participated in a qualitative research study to create a distance vision subjective manifest refractometry rubric by identifying the key steps of the procedure, determining performance levels of competency, and creating behavioral descriptors for each procedural step and level of competency. The rubric was reviewed by a total of 32 international subject-matter experts from 18 countries and from various roles, to ensure content validity. 

Results: This internationally-standardized rubric is a content-valid competency assessment tool that could be applied globally to teach and assess distance vision manifest subjective refractions.

Conclusions: It was important to collaborate with international subject-matter experts to develop a rubric that was specific enough to ensure proper steps are being followed, yet flexible enough in the wording that is can be adapted to regional/local methods. 

Additional rubrics for near vision refraction, cycloplegic refraction, and other ophthalmic skills are being considered.

“Having Your Back While Keeping Your Gaze Forward”: Longitudinal Coaching Relationships and Their Effect on Feedback Processes - A Grounded Theory Study

Priya Jain
Ann & Robert H. Lurie Children's Hospital


Purpose: Health professions educators have begun shifting their views of feedback from one-directional giving of information to bi-directional dialogue that also incorporates how learners receive and process that information. This shift foregrounds the role of educational alliances. This study explores how longitudinal relationships between trainees and faculty coaches influence feedback processes and what factors help or hinder these relationships and processes.

Methods: The authors used constructivist grounded theory methodology and performed semi-structured interviews of pediatric fellowship trainees, faculty coaches and program directors from two pediatric subspecialty fellowship training programs. Both programs had previously implemented longitudinal Clinical Coaching Programs. Interviews were conducted between 2019 and 2020.

Results: The authors identified significant positive influences of effective longitudinal relationships between faculty coaches and trainees. These influences shaped both enactments of the coaching role and feedback processes that fostered their success in key ways, A primary success factor was reflected in the individualized tailoring of feedback and coaching processes based on a growth mindset, trainee feedback preferences, and knowledge of prior coaching conversations. In addition, multiple factors hindered successful coaching relationships. These barriers included (a) inadequate preparation for coaching conversations, (b) lack of specific feedback provided by the coach, and (c) coaches’ failure to encourage trainees to develop individualized improvement goals.

Conclusions: Simply providing feedback to trainees no longer suffices in today’s health professions education landscape. Feedback processes embedded within coaching programs enable faculty coaches to develop strong longitudinal relationships with trainees and to implement individualized coaching techniques. These growth-oriented educational alliances support trainees in fundamental ways, namely in processing feedback and guiding development of personalized learning goals. Clinical training programs can help their trainees foster invaluable skills that promote life-long learning and continuous self-improvement.

Camp for Children with Diabetes: A Multi-Disciplinary Medical Staff Simulation Training Curriculum

Erin Atwood

Cincinnati Children's Hospital

Appropriately trained medical staff and a medically safe environment are critical at diabetes camps. We designed a simulation-based curriculum for training medical staff to manage common diabetes scenarios in a camp environment. Camp medical staff from a variety of disciplines participated in three simulation scenarios focused on responding to diabetes emergencies (hypoglycemia and hyperglycemia with ketosis) and addressing use of insulin pumps. Scenarios were developed utilizing the conceptual framework of cognitive load theory and purposefully included background distractions to mimic a camp environment. Participants completed demographic surveys (including profession, prior experience in diabetes) and workload assessments. Twenty-five medical staff completed the surveys; most were female, aged 25-34 years, and physician trainees (residents, fellows). The majority of medical staff (N=16) had some diabetes training prior to camp orientation. However, very few had experience with or training in diabetes technology or emergencies that required the administration of insulin or glucagon. Workload assessments, completed at the beginning and end of camp by the simulation participants revealed a statistically significant decrease in temporal demand (p = 0.03) felt by the medical staff. Our simulation curriculum addressed a knowledge gap related to emergency management of diabetes and understanding of diabetes technology. It also was successful in demonstrating a decrease in temporal load or time demands reported by the participants. Next steps include comparative studies of simulation vs. standard orientation on medical staff knowledge and cognitive load, and evaluation of camper outcomes.