Do The Licensing Examinations (USMLE Step 1 and Step 2 Clinical Knowledge) Measure Something Meaningful?

USMLE: Probably, according to this study of more than 160,000 practicing physicians who graduated from US MD-granting schools between 1994 and 2006.  The authors looked for correlations between a physician’s examination scores and their likelihood of receiving disciplinary action from a state medical board.  Overall, only 1.3% of these physicians had received such an action.  For both exams, higher scores were associated with a lower risk of receiving disciplinary action, but once Step 2 scores were included in the model, Step 1 scores had no further contribution to the effect.  The effect was moderate:  a 1 SD increase in Step 2 score (about 23 points) was associated with a 25% lower risk of receiving an action.  It would have been useful to know the reasons for actions taken and whether higher board scores reduced some classes of actions (e.g. clinical incompetence citations) more than others (e.g. professionalism citations). — Laura Willett, MD

Cuddy MM, Young A, Gelman A, Swanson DB, Johnson DA, Dillon GF, Clauser BE. Exploring the Relationships Between USMLE Performance and Disciplinary Action in Practice: A Validity Study of Score Inferences From a Licensure Examination.
Acad Med. 2017

Link To Article

 

Residency Program Directors Like The Idea Of “Flipped Classroom” Active Learning, But Don’t Use It Very Often

Flipped Classroom: Only 56% of internal medicine program directors responded to survey questions regarding attitudes towards and use of “flipped classroom” techniques in their program, but their responses are likely somewhat representative.  Overall attitudes were positive, averaging about 4 on a 5-point scale, with slightly more positive perceptions towards in-class application exercises vs. pre-class readings or other preparation.  However, only 12% of program directors reported using these techniques “somewhat” or “very often”, and 59% reported using them “somewhat rarely”, “very rarely”, or  “never”.  Unsurprisingly, program directors with more positive attitudes reported greater use of these techniques. — Laura Willett, MD

Wittich CM, Agrawal A, Wang AT, Halvorsen AJ, Mandrekar JN, Chaudhry S, Dupras DM, Oxentenko AS, Beckman TJ.Flipped Classrooms in Graduate Medical Education: A National Survey of Residency Program Directors. Acad Med. 2017 Jun.

Link To Article

How Residents Die

Risk: In this survey of deaths of residents from all ACGME programs, malignancy and suicide were the most common causes of death.  Hearteningly, death rates from all causes and from suicide were substantially lower in residents than in the general population.  This is in distinction from suicide rates in practicing physicians, which are higher than the general population.  The time of highest risk for suicide was the first quarter of internship year.  The report reinforces the need for behavioral health care for our residents; fairly common causes of death included suicide, accidents, deaths of “ill-defined or undetermined intent”, and accidental poisoning. — Laura Willett, MD

Yaghmour NA, Brigham TP, Richter T, Miller RS, Philibert I, Baldwin DC Jr, Nasca TJ.Causes of Death of Residents in ACGME-Accredited Programs 2000 Through 2014: Implications for the Learning Environment.
Acad Med. 2017 May

Link To Article

Are Written Words A Reliable Way To Assess Residents?

Assessment: It looks like it.  In this insightful study, Canadian researchers sent collections of comments from evaluators of two cohorts of nearly 50 interns in one internal medicine program to faculty in several other programs.  Each faculty person was asked to rank-order 16 interns according to the written comments only; associated numerical grades were hidden from the rankers.  A high degree of reliability was found (G about 0.80) with as few as two independent faculty members viewing only the first 3 written comments for an individual intern, obtained over the intern’s first 4 months of training. Maybe qualitative comments are the “gold standard”? — Laura Willett, MD

Ginsburg S, van der Vleuten CP, Eva KW., The Hidden Value of Narrative Comments for Assessment: A Quantitative Reliability Analysis of Qualitative Data. Acad Med. 2017 Apr.

Link To Article

 

When The Monday Before Match Day Is Not A Happy Day

Match: No research, and only a smattering of data, but this article is required reading for faculty counseling graduating MD students from the US with difficulty matching into residency.  A sizable percentage (6.2%) of US seniors did not gain a position during the 2016 primary Match; nearly half of these attained a position during the supplementary program (SOAP) in the days shortly before Match Day.  For those who do not (615 students in the 2016 Match), the authors review a number of options, including delay of graduation, obtaining a position outside of SOAP (fairly high success), re-entering the Match the following year in the same or different specialty (also fairly high success), pursuing interim activities to improve Match success, or pursuing a non-clinical career.  A systemic problem is that there is no centralized clearinghouse of open residency positions, which may occur off-cycle.  The authors also address more specific issues for school administrators such as policies/practices regarding delay of graduation, Dean’s Letter update, leave of absence from rotations following Match failure, and support for students post-graduation. — Laura Willett, MD

Bumsted T, Schneider BN, Deiorio NM. Considerations for Medical Students and Advisors After an Unsuccessful Match. Acad Med. 2017 March

Link To Article

 

Don’t Know Much About….Feedback

Feedback: This scoping review which describes what has been published in English regarding feedback to healthcare learners will be of interest mostly to researchers and those looking for a comprehensive view of the field.  The bulk of publications were from English-speaking countries and focused on medical students or residents in those countries.  Of publications with a specialty focus, a majority involved either internal medicine or surgery.  The overall level of the literature seems to be in a nascent stage.  For example, even though a majority of articles described a new approach to feedback, only 7.5% of articles involved randomization of learners.  Likewise, in the large number of articles that evaluated a feedback intervention, 57.7% confined that evaluation to the learners’ reaction to feedback, rather than measuring more important outcomes such as changes in care or behavior.  “High-level, evidence-based educational recommendations for feedback are lacking.” — Laura Willett, MD

Bing-You R, Hayes V, Varaklis K, Trowbridge R, Kemp H, McKelvy D. Acad Med. 2017 Feb 7. Feedback for Learners in Medical Education: What Is Known? A Scoping Review.

Link To Article

 

 

Faculty Development For Community-Based Faculty

Faculty Development: In this survey of North American family medicine clerkship directors, researchers focused on the use of community-based preceptors and how they are developed and rewarded. Sixty-nine percent of schools reported heavy reliance on community-based preceptors for their family medicine clerkship instruction.  Interestingly, 100% of Canadian clerkship directors reported giving direct monetary payments to their community-based preceptors, as opposed to 28% of US directors.  Among schools that paid their preceptors, the average payment was $263 per week.  Schools which paid their preceptors were much more likely to require those preceptors to participate in formal faculty development.  The most common form of formal faculty development was the in-person session, despite the largest barriers to participation being reported as preceptor availability and geographic distribution.  Various forms of distance learning were also utilized.  There was little evidence of rigorous needs assessment or evaluation of effectiveness for most community-based faculty development programs. — Laura Willett, MD

Drowos J, Baker S, Harrison SL, Minor S, Chessman AW, Baker D., Faculty Development for Medical School Community-Based Faculty: A Council of Academic Family Medicine Educational Research Alliance Study Exploring Institutional Requirements and Challenges. Acad Med. 2017 Feb 21.

Link To Article