Does A Milestone-Based Evaluation Have Better Discriminatory Ability? Yes and No.

evaluations    Residency programs across the country have recently transitioned to a Milestone-based evaluation form in response to ACGME’s call to measure and report resident performance via education milestones. Prior to this change, many evaluation systems relied on the Dreyfus framework of assessment, which describes a learner as a novice, advanced beginner, competent, proficient, or expert. In this before/after study from a university-based internal medicine residency program, investigators compared scores on resident evaluations between a Dreyfus-based model of evaluation and a new Milestone-based evaluation. Both evaluation systems assessed resident performance in the 6 ACGME competency domains using a 5-point scale: for the Dreyfus model, 1 = beginner, 2 = advanced beginner, 3 = approaching competency, 4 = competent, and 5 = advanced competent; for the Milestone-based model, a 5-point scale was anchored by performance expected at a given level of training, where 1= 0-3 months, 2 = 6 months, 3 = 12 months, 4 = 18-24 months, 5 = 30-36 months. While the study investigators report better discriminatory ability with the Milestone-based evaluation as demonstrated by larger separation in scores among the 3 PGY classes with wider use of the 5-point scale, they did not report discriminatory ability within PGY classes. In fact, standard deviations around mean scores were consistently smaller with the Milestone-based evaluation compared to the Dreyfus-based evaluation. It’s quite plausible that when a scale is anchored by level of training, evaluators may feel “free” to use lower ratings for early PGY-1’s. Similarly, evaluators may have felt compelled to use higher ratings for graduating residents, though this was not addressed by the study. — Sarang Kim, MD

Friedman KA, Balwan S, Cacace F, Katona K, Sunday S, Chaudhry S. Impact on house staff evaluation scores when changing from a Dreyfus- to a Milestone-based evaluation model: one internal medicine residency program’s findings. Med Educ Online. 2014 Nov 24;19:25185.

Link To Article

This entry was posted in Graduate Medical Education and tagged , , . Bookmark the permalink.