University of Kentucky / Albert B. Chandler
Background: Though Item Response Theory (IRT) is a well-established method to analyze response data, the utility of IRT in clinical skill assessment is not well-described. Using the procedure simulation test data whose standards were determined by the Angoff method, we explored (1) the psychometric properties of the test and (2) how different passing scores change the interpretations of the test.
Methods: Our study was conducted at a single US medical school where 3rd-year medical students are required to pass 10 procedural skill simulations. The unidimensional partial credit model was applied to the data of 131 students whose total scores for each procedure were recorded. We mapped person locations and thresholds on the same continuum (i.e., the item-person map) which allowed us to assess how the test performance would change by different passing scores (i.e., Angoff, +1, +2, and perfect scores).
Results: The item-person map elucidated a gap between student abilities and item difficulties (thresholds). The Angoff scores were too easy to differentiate high- from low-performers. Adding 1 point to the Angoff scores resulted in the largest variance, that would allow the test to assess a wider range of skills. This procedural skill test however was not suited to assess a wide range of skills unless we make some items more difficult and appropriate for the target populations.
Conclusions: This study illustrates the application of IRT in clinical skill assessment. The results showed that IRT was useful to refine the test itself and the passing scores.
