Can You Observe Me AFTER Him (or Her)?  Prior Observations Affect Graders of High-Stakes Evaluations

evaluation Replicating the results of experiment, investigators found that examiners’ ratings of candidates in two very different high-stakes behavioral testing situations were affected by the performance of prior candidates.  The first group of scores were from 2,272 takers of a UK 16-station OSCE required for any medical school graduate who has been out of medical school for more than 2 years and who wishes to pursue subspecialty training.  The second were the multiple mini-interview (MMI) scores of 3,016 applicants to the University of Alberta Medical School.  Consistent negative correlations were found between the index score and those of preceding examinees.  That is, a candidate was graded somewhat higher if he or she was preceded by less-ranked candidates, and vice versa.  The effect accounted for 5% to 11% of total score variance.  This appears to be a fairly robust effect.  Now the difficulty will be in deciding how to correct or ameliorate it. — Laura Willett, MD

Yeates P1, Moreau M, Eva K. Are Examiners’ Judgments in OSCE-Style Assessments Influenced by Contrast Effects? Acad Med. 2015 Jan 27.

Link To Article

This entry was posted in Undergraduate Medical Education and tagged , , . Bookmark the permalink.