University of Cincinnati
Purpose: To develop and establish validity evidence for a novel tool to assess trainee admission notes that would assess key components of a note, provide formative feedback, and provide global assessments mapped to the ACGME milestone framework.
Method: An admission note assessment tool was developed through an iterative, consensus building process utilizing local experts. Pilot testing was performed comparing ratings by the supervising attending to ratings by study team raters to evaluate if the supervising attending needs to complete note assessment. The finalized tool was then piloted with attendings from other institutions to evaluate feasibility of tool implementation at other institutions.
Results: The final tool consists of 16 discrete checklist items and two global assessment items. Pilot testing demonstrated overall high rater agreement between the supervising attending and study team raters. The study generated validity evidence in the domains of content, response process, and internal structure for use of the tool in rating admission notes.
Conclusions: The admission note assessment tool uniquely provides assessment of individual components of a note, including billing criteria, targets chart “bloat”, and allows for formative feedback while providing global assessments mapped to the ACGME milestone framework. The tool can be used to assess trainee admissions notes by any attending and at any time after note completion with minimal rater training. This suggests the feasibility of implementing routine note assessment in a training program and ability to use a single tool for multiple assessment purposes.
