Assessment: It looks like it. In this insightful study, Canadian researchers sent collections of comments from evaluators of two cohorts of nearly 50 interns in one internal medicine program to faculty in several other programs. Each faculty person was asked to rank-order 16 interns according to the written comments only; associated numerical grades were hidden from the rankers. A high degree of reliability was found (G about 0.80) with as few as two independent faculty members viewing only the first 3 written comments for an individual intern, obtained over the intern’s first 4 months of training. Maybe qualitative comments are the “gold standard”? — Laura Willett, MD
Ginsburg S, van der Vleuten CP, Eva KW., The Hidden Value of Narrative Comments for Assessment: A Quantitative Reliability Analysis of Qualitative Data. Acad Med. 2017 Apr.