Using EPA’s For Competency-Based Assessment Of Medical Students In The Internal Medicine (IM) Clerkship

Assessment: Alliance for Academic Internal Medicine Undergraduate Medical Education (UME) Task Force proposes a model for competency-based assessment in IM clerkships, informed by results of a survey of clerkship directors who were asked to identify high priority EPA’s (entrustable professional activities) for students in the IM clerkship. Six EPA’s were identified as important for assessment in the IM clerkship and were related to accuracy in obtaining a history and physical, importance of verbal and written communication, interpretation of diagnostic studies, and generating differential diagnoses. Subdivisions under these EPA’s were also suggested, with use of an entrustment/supervision scale anchored by “not allowed to practice,” “allowed under full supervision,” and “allowed with on-demand supervision.”

What the authors propose is a potentially useful model for competency-based assessment in the IM clerkship that would also align UME assessment with how GME assessment is currently being done. However, their proposal is not quite ready for use in the current educational environment. A formidable challenge to implementation of such an assessment is the larger paradigm shift that would need to take place to replace clerkship grades with a dichotomous scale of meets standards or does not meet standards. How likely that is to happen in the next few years is hard to say but this is a topic that is being actively debated in the educational community. So stay tuned…
–Sarang Kim, MD

Fazio SB, Ledford CH, Aronowitz PB, Chheda SG, Choe JH, Call SA, Gitlin SD, Muntz M, Nixon LJ, Pereira AG, Ragsdale JW, Stewart EA, Hauer KE. Competency-Based Medical Education in the Internal Medicine Clerkship: A Report From the Alliance for Academic Internal Medicine Undergraduate Medical Education Task Force. Acad Med. 2017 Sep

Link to Article

Posted in Graduate Medical Education, Undergraduate Medical Education | Tagged , , ,

Does Self-Regulated Test-Prep Work To Improve USMLE Step 1 Scores?   

USMLE: Probably, and it is likely most helpful for the students with lower-than-median MCAT scores, according to this non-randomized intervention for pre-clinical medical students.  All students in one class were provided free access to a commercial NBME-style question bank and their access to questions was tracked.  Higher utilization of test questions was associated with higher objective achievement including USMLE 1 scores for all students, but the gain was much higher in those students with lower-than-average MCAT scores.  For example, in going from 0 to nearly 1,500 questions accessed (the range seen in this group), students with the median MCAT score of 30 were projected to raise their USMLE 1 score by about 20 points.  Students in the lowest MCAT group were projected to have a rise of about 40 points, while those in the highest group experienced a minimal rise. The authors suggest that this access may “benefit all students while conferring relatively greater benefit to students who may enter medical school with greater academic risk or less well-developed standardized test-taking skills.”  Obvious confounders are student persistence and work ethic.  It would be useful to find out exactly how the students are using these questions – e.g. mostly to “cram” for exams, or for paced re-exposure to previously-learned material. — Laura Willett, MD

Baños JH, Pepin ME, Van Wagoner N. Class-Wide Access to a Commercial Step 1 Question Bank During Preclinical Organ-Based Modules: A Pilot Project. Acad Med. 2017 Aug 16.

Read Article


Posted in Undergraduate Medical Education | Tagged , , ,

Interns And More Senior Residents Fail To Meet Expectations In Different Ways

Milestones: In this study involving 20% of pediatric residency training programs in the US, milestone evaluations of residents rated marginal or unsatisfactory (M/U) were compared  to those of residents rated satisfactory.  Overall, fewer than 2% of residents were rated less than satisfactory, with interns (first-year residents) and international medical graduates slightly more likely to be categorized in this way.  On a 5-point milestone scale, M/U interns rated significantly lower in every subcompetency, with average ratings 0.60-0.97 below their “satisfactory” peers.  Two subcompetences discriminated very well between the groups of interns:  organization/prioritization and transfers of care.  With increasing resident seniority, the differences in average ratings between satisfactory and M/U residents became much more variable with the subcompetency, with the largest differences found in professionalism, trustworthiness, and transfers of care. — Laura Willett, MD

Li ST, Tancredi DJ, Schwartz A, Guillot A, Burke A, Trimm RF, Guralnick S, Mahan JD, Gifford KA; Association of Pediatric Program Directors (APPD) Longitudinal Educational Assessment Research Network (LEARN) Validity of Resident Self-Assessment Group. Identifying Gaps in the Performance of Pediatric Trainees Who Receive Marginal/Unsatisfactory Ratings. Acad Med. 2017 Jun

Link To Article


Posted in Graduate Medical Education | Tagged , , ,

How To Not Get Published, Or Even Peer-Reviewed

Publishing: This is required reading for faculty who want, or need, to publish medical education research articles.  Researchers describe the pre-peer review editorial process at Academic Medicine, during which 65% of submitted manuscripts are rejected, either directly by the editor-in-chief or by associate editor review.  Free text comments from associate editors regarding 369 manuscripts selected for expedited rejection were analyzed qualitatively.  The average rejected article had 3.11 reasons for rejection, with the reasons falling into 9 major themes.  Themes represented by more than 30% of rejected manuscripts were:  “ineffective study question or design” 92%; “suboptimal data collection process” 49%; “weak discussion and/or conclusions” 37%; unimportant or irrelevant topic to the journal’s mission” 37%; “weak data analysis and/or presentation of results” 33%.  Many of these objections could likely be avoided by better planning during the study design phase. — Laura Willett, MD

Meyer HS, Durning SJ, Sklar D, Maggio LA. Making the First Cut: An Analysis of Academic Medicine Editors’ Reasons for Not Sending Manuscripts Out for External Peer Review. Acad Med. 2017 Aug.

Link To Article

Posted in Graduate Medical Education, Undergraduate Medical Education | Tagged , , ,

Do The Licensing Examinations (USMLE Step 1 and Step 2 Clinical Knowledge) Measure Something Meaningful?

USMLE: Probably, according to this study of more than 160,000 practicing physicians who graduated from US MD-granting schools between 1994 and 2006.  The authors looked for correlations between a physician’s examination scores and their likelihood of receiving disciplinary action from a state medical board.  Overall, only 1.3% of these physicians had received such an action.  For both exams, higher scores were associated with a lower risk of receiving disciplinary action, but once Step 2 scores were included in the model, Step 1 scores had no further contribution to the effect.  The effect was moderate:  a 1 SD increase in Step 2 score (about 23 points) was associated with a 25% lower risk of receiving an action.  It would have been useful to know the reasons for actions taken and whether higher board scores reduced some classes of actions (e.g. clinical incompetence citations) more than others (e.g. professionalism citations). — Laura Willett, MD

Cuddy MM, Young A, Gelman A, Swanson DB, Johnson DA, Dillon GF, Clauser BE. Exploring the Relationships Between USMLE Performance and Disciplinary Action in Practice: A Validity Study of Score Inferences From a Licensure Examination.
Acad Med. 2017

Link To Article


Posted in Graduate Medical Education | Tagged ,

Residency Program Directors Like The Idea Of “Flipped Classroom” Active Learning, But Don’t Use It Very Often

Flipped Classroom: Only 56% of internal medicine program directors responded to survey questions regarding attitudes towards and use of “flipped classroom” techniques in their program, but their responses are likely somewhat representative.  Overall attitudes were positive, averaging about 4 on a 5-point scale, with slightly more positive perceptions towards in-class application exercises vs. pre-class readings or other preparation.  However, only 12% of program directors reported using these techniques “somewhat” or “very often”, and 59% reported using them “somewhat rarely”, “very rarely”, or  “never”.  Unsurprisingly, program directors with more positive attitudes reported greater use of these techniques. — Laura Willett, MD

Wittich CM, Agrawal A, Wang AT, Halvorsen AJ, Mandrekar JN, Chaudhry S, Dupras DM, Oxentenko AS, Beckman TJ.Flipped Classrooms in Graduate Medical Education: A National Survey of Residency Program Directors. Acad Med. 2017 Jun.

Link To Article

Posted in Graduate Medical Education | Tagged

How Residents Die

Risk: In this survey of deaths of residents from all ACGME programs, malignancy and suicide were the most common causes of death.  Hearteningly, death rates from all causes and from suicide were substantially lower in residents than in the general population.  This is in distinction from suicide rates in practicing physicians, which are higher than the general population.  The time of highest risk for suicide was the first quarter of internship year.  The report reinforces the need for behavioral health care for our residents; fairly common causes of death included suicide, accidents, deaths of “ill-defined or undetermined intent”, and accidental poisoning. — Laura Willett, MD

Yaghmour NA, Brigham TP, Richter T, Miller RS, Philibert I, Baldwin DC Jr, Nasca TJ.Causes of Death of Residents in ACGME-Accredited Programs 2000 Through 2014: Implications for the Learning Environment.
Acad Med. 2017 May

Link To Article

Posted in Graduate Medical Education | Tagged ,