US Medical Education By The Numbers (LCME and ACGME Divisions)

Medical Education: JAMA again presents a deep dive into the numbers and characteristics of learners in MD-granting US medical schools and in US residency programs accredited by the ACGME, which is now encompassing most of the programs previously accredited by the Association of American Colleges of Osteopathic Medicine (AACOM).

Regarding MD-granting medical schools:  The number of medical students per 100,000 state residents is quite variable among the states which have medical schools, from a low of 11.6/100,000 in Arizona to 75.6/100,000 in Vermont.  After many years of stable total enrollment of about 70,000 until 2008, the total enrollment has risen to about 86,000 in the most recent data, with close to even numbers of women and men.  Fifty-eight percent of schools noted “increased difficulty finding and retaining inpatient clinical placements” for their clerkships, primarily due to competition for sites from other medical or health-professions schools.

Regarding ACGME-accredited residency and fellowship programs:  From 2013-14 to 2018-19, the number of trainees in these programs expanded from about 117,000 to about 136,000.  This included increases of about 1,000 international medical graduates (IMGs), about 8,000 US MD graduates, and about 10,000 US osteopathic graduates.  This reviewer assumes that much of the latter increase is due to the incorporation of many programs previously certified by AACOM.  Learners in these residency programs come primarily from US MD-granting schools (62.7%), DO-granting schools (14.2%), and international medical schools (23.0%).  These learner types are not evenly spread among the specialties.  IMGs are disproportionately represented in neurology, pathology, and internal medicine and its specialties.  DO recipients are disproportionately represented in emergency medicine, family medicine, and physical medicine/rehabilitation.  Among IMGs, 26% are native US citizens, 12% are naturalized US citizens, and 15% are permanent residents.  The majority of the remainder are visa holders.  The racial and ethnic composition of resident physicians is compared below to the composition of the US population (2010 census data per Wikipedia).  From least to most represented proportionately, they are:  American Indian/Alaskan native 0.11, Black 0.43, Hispanic 0.49, Pacific islander/native Hawaiian 0.50, White 0.78, Multiracial/other/unknown 1.2, and Asian 5.5.  Estimates of proportions for American Indian/Alaskan native and Pacific islander/native Hawaiian are based on low numbers and likely less reliable. — Laura Willett, MD

Barzansky B, Etzel SI, Medical Schools in the United States, 2018-2019.
JAMA, 2019 Sep 10;322(10):986-995.
Link To Article 

Brotherton SE, Etzel SI, Graduate Medical Education, 2018-2019
JAMA. 2019 Sep 10;322(10):996-1016.
Link To Article

Mid-Level MCAT Scorers Increase Diversity in Medical Schools

Diversity: Medical schools seeking greater diversity in their classes may want to look at data from the 2017 medical school applicants who scored in the middle third of the MCAT range.   Compared to students who scored in the upper-third on the exam, students with mid-third scores show a greater tendency to be the first in their family to graduate from college, to have parents working in the service industry, to have grown up in rural or medically-under served parts of the country, to be non-native English speakers and to be underrepresented based on race/ethnicity.  The authors note that ​in a companion piece in Academic Medicine, success in  medical school can be achieved by students scoring along a range of MCAT scores.  Applicants who score in the mid range of the MCAT score range advance to the second year of medical school nearly as frequently as upper-tier students (95% vs. 98%).    Given the ​importance of meeting the needs of an increasingly diverse US population, the authors suggest medical schools continue​ to use holistic review practices and to look beyond accepting only students with high MCAT scores.  — DR MERL Editors 

Terregino, CA, Saguil A, Price-Johnson T, Anachebe NF, Goodell K. The Diversity and Success of Medical School Applicants with Scores in the Middle Third of the MCAT Score Scale . published ahead of print August 13, 2019]. Acad Med. doi: 10.1097/ACM.0000000000002941.

Link To Article

Author Interview
The editors of DR MERL spoke with Carol Terregino, MD about her recent publication in Academic Medicine.

What got you interested in the topic of MCAT scores and diversity? 

Terregino: “I have been working in medical school admissions for over 20 years.  Robert Wood Johnson Medical School has been in the business of recruiting and supporting students from diverse groups well before terms such as holistic admissions and student retention became part of the vocabulary of medical student selection.  I have had some of my most meaningful professional experiences in collaboration with staff and faculty from across the nation by participation in research groups at the Association of American Medical Colleges. I was part of the Innovation Lab Working Group of the MR5 Committee (development of MCAT 2015) and H. Liesel Copeland PhD, assistant dean for admissions and medical education and I are currently a part of the MCAT Validity Working Group, made of 16 medical schools representing all regions of the nation, public and private institutions, with missions ranging from research intensive to community based.  We meet semiannually for this multiyear study and have the opportunity to debate important ideas in the world of medical school admissions and diversity with colleagues of similar values.

The current paper is a collaborative work by academic medicine faculty from USUHS, Morehouse School of Medicine, Boston University, Arizona College of Medicine Tucson.  We were committed to the following ideas:

• • • The evidence that MCAT scores predict academic performance in medical school and on licensing examinations is strong.
    • In every class we admit, we see variability in the academic performance of students with similar MCAT scores.
    • Students with more modest MCAT scores can and do outperform their classmates with higher MCAT scores.
    • Success depends upon the academics, attributes, and experiences these applicants bring.
    • Success also depends upon our ability to provide a learning environment in which students from different backgrounds and academic trajectories could thrive.

We all need to look beyond the numbers to diversify our classes and meet needs of our populace.”

 

Were you surprised by your findings?  

Terregino: “Given our history meeting the needs of medical students across a range of MCAT scores, I was not surprised with the success rate across the score range, but I must say I was very gratified to see that the new MCAT performed in this way.  I am surprised that there has been no public comment regarding the relative indictment of methodology of medical school rankings and schools reporting pressure to accept only applicants with high scores.”

 

What changes would you like to see come from your article?

Terregino: “I am hoping that as each school defines who comprises their underrepresented in medicine groups, they search for these applicants and review each application thoughtfully, acknowledging how life experiences and personal attributes add context to a given MCAT score-knowing that success can be obtained with scores beneath the highest ones.  There are absolute gems  in this cohort who should not be overlooked.”

 

What’s next for you in terms of medical education research?

Terregino: “My focus in medical education is ensuring that when any patient looks into the eyes of one of our graduates, they find a doctor adept in communicating and developing a therapeutic alliance, well-versed in all realms of professional communication and teamwork, and one with outstanding clinical reasoning and critical thinking skills. My research projects range from admissions, to developing a new communication rating scale to teaching and measuring teamwork skills.”

Do Licensing Exam Scores Predict Important Outcomes?

Exam Scores:  It looks like it, according to this study.  Researchers looked for a correlation between scores on the COMLEX (osteopathic physician licensing exams given in a similar sequence to USMLE exams) and major actions against physicians made by state licensing boards.  There was a strong negative correlation between the score and the chance of an adverse action, particularly for the Step 3 score; at least a trend was seen for scores on all of the exams. — Laura Willett, MD

Roberts WL, Gross GA, Gimpel JR, Smith LL, Arnhart K, Pei X, Young A., An Investigation of the Relationship Between COMLEX-USA Licensure Examination Performance and State Licensing Board Disciplinary Actions. Acad Med. 2019 Oct 15

Link To Article

Are ACGME Work Hour Restrictions Effective? Reports from Interns and Program Directors

Duty Hours: Libby Zion died under the care of house officers in 1984 during a time when there was a 36-hour work shift policy for residents and interns. To minimize medical errors caused by fatigue, the ACGME created new restrictions on duty hours (first in 2003 and then revised in 2011) that are currently in use. However, with no clear evidence of improved patient outcomes and complaints by program directors of poorer training, the Comparativeness Effectiveness of Models Optimizing Patient Safety and Resident Education study (aka iCOMPARE) was developed to evaluate how these changes affect trainees. Funded by the National Institutes of Health (NIH), 63 residency programs were randomly assigned to standard duty-hour policies (maximum 16-hour shifts with required time off between shifts) or to flexible policies which didn’t impose any time constraints. Both groups maintained a maximum of 80 hours of work per week, an average of one day off every seven days, and overnight call no more frequent than every 3 nights.

Most of the primary endpoints did not differ between the two groups. There was no significant difference in time spent in direct patient care, education time, scores on in-training exams, trainee perception of a balance between clinical demands and education, or faculty perception of excess workload. However, interns in the flexible programs reported a more negative effect on numerous training and personal experiences, including patient safety; research, educational, and teaching opportunities; and morale, job satisfaction, time with family and friends, and overall well-being. Interestingly, there was no difference between the two groups in regard to burnout, using the Maslach Burnout Inventory. On the other hand, in the flexible programs interns reported better continuity of care and program directors reported less dissatisfaction with the learning environment, notably with the ability to give feedback and the frequency of handoffs.

While the study leaves us with a great concern over duty hours effect on trainee’s perception of their training and wellness, it doesn’t tell us if we are improving patient care and limiting future cases like that of Libby Zion. – David A. Cohen, MD

Desai SV, Asch DA, Bellini LM, Chaiyachati KH, Liu M, Sternberg AL, Tonascia J, Yeager AM, Asch JM, Katz JT, Basner M, Bates DW, Bilimoria KY, Dinges DF, Even-Shoshan O, Shade DM, Silber JH, Small DS, Volpp KG, Shea JA; iCOMPARE Research Group. Education outcomes in a duty-hour flexibility trial in internal medicine. NEJM. 2018;378:1494–508. 

Link To Article

 

An MD In 3 Years For Some – NYU’s Roadmap

Pathway:  This descriptive article of New York University’s program gives something of a roadmap to schools which are considering offering a similar option to their students.  The authors describe a much more flexible program than many other 3-year tracks.  All students admitted to the overall 4-year program can apply for the 3-year track, and students can opt-in at multiple different times including as late as the spring of second year.  Students can opt-out of the 3-year track and re-enter the 4-year track, or be required to re-enter the 4-year track because of academic or professionalism concerns, at any time.   This is made possible by a curriculum which is shared by students in both tracks with 18 months of pre-clinical instruction followed by 12 months of required clerkships.  Unlike many 3-year programs in which students must agree to go into a specified primary care residency after medical school completion, students can apply at entry to any of NYU’s residency programs, which save about 20% of their slots for students in the 3-year program.  Students in the program are also able to change their specialty, although few availed themselves of this opportunity.   The authors describe fairly intensive involvement of residency program directors in selection and multiple mentors and advisors for the students.  So far, the residency directors of these new residents are happy with their performance.   Some questions unanswered by the current publication are:  Are students in the 3-year- program much better prepared than other NYU students in any identifiable way?  How many students who applied to the 3-year track were turned down, and for what reasons?  What would happen if a residency program director becomes very uncomfortable with the promise of a guaranteed Match spot to a particular student?  Would this program work as well for a school with less prestigious residency programs?  Longer-term outcomes will be interesting. — Laura Willett, MD

Cangiarella J, Cohen E, Rivera R, Gillespie C, Abramson S. Evolution of an Accelerated 3-Year Pathway to the MD Degree: The Experience of New York University School of Medicine Acad Med. 2019 Oct 1.

Link To Article

Are Entrustment Ratings Reliable In The Era of EPA’s?

Entrustable Professional Activities: The short answer is:  not really.  In this study looking at reliability of entrustment ratings, the authors examined faculty ratings of “observable practice activities (OPA)” over 36 months of clinical rotations for internal medicine residents at a single US academic institution. OPA’s were assessed using an entrustment scale ranging from 1=not entrusted to perform activity even with direct supervision, to 5=entrusted to perform activity independently at an aspirational level. The ratings were analyzed using both a time-specific and a longitudinal generalizability theory analyses to assess reliability of entrustment ratings and sources of variance, including maturation effect, an important factor when looking at improvement of skills over time.  With 1-2 faculty raters per resident per month and a faculty evaluation completion rate of 80% (now that’s aspirational!), over 166,000 OPA ratings by 395 faculty for 253 residents were assessed. The results are disappointing and familiar: there is only fair reliability (0.40 for a month of evaluations, 0.63 over the whole 36-month duration of residency), and the largest source of variance is the rater (37%), then the resident (19%). A decision study showed that doubling the number of raters and assessments each month (to 2-4 evaluations per resident per month) would increase the reliability over 36 months to 0.76, a number that is getting closer to an acceptable range for making high stakes decisions.

Given these findings, how does a program, especially ones with even fewer evaluations per resident, make entrustment decisions?  First, one must place more effort in rater training.  Second, it is useful to recognize that some inconsistency by raters may reflect true differences in skill level given varying context, case-mix, etc, and aiming for perfect reliability is neither possible nor sensible. Finally, narrative comments, which were not addressed in this study, likely capture rich data that may not be described by ratings alone. — Sarang Kim, MD

Kelleher, Matthew MD, MEd; Kinnear, Benjamin MD, MEd; Sall, Dana MD, MEd; Schumacher, Daniel MD, MEd; Schauer, Daniel P. MD, MSc; Warm, Eric J. MD; Kelcey, Ben PhD A Reliability Analysis of Entrustment-Derived Workplace-Based Assessments Academic Medicine: September 17, 2019

Link To Article

Who Gets Honors?

Honors:  It may depend more on where you are than who you are.  This look at the “dean’s letters” or medical student performance evaluations (MSPEs) from 97% of US-based MD-granting schools shows some improvements in transparency from previous surveys, but continued enormous variability in grading systems and creeping grade inflation.  Researchers from Mayo clinic looked at MSPEs from applicants to two of their residency programs, with the unit of analysis being the school generating the MSPE.  These schools had 19 distinct grading systems for clinical clerkships, with the number of tiers ranging from 2 to 7, median 4.  Contrary to current MSPE guidelines, 10% provided no grade distribution information.  Among MSPEs which outlined grade distributions, 33% percent of clerkship grades were the highest grade available (usually Honors), which has gone up steadily in various surveys from 22% approximately 20 years ago.  Well fewer than 1% of grades were failing.  Predictors of a top-tier grade were: 4- or 5-tier grading system (vs. lower or higher number of tiers), psychiatry clerkship, Southern region, and medical school in the top 20 research schools in US New and World Report rankings (see recent commentary on these rankings).  Regarding the MSPE summarizing statement, 27% of letters gave no indicator overall ranking, 42% used up to 32 different adjectives to describe the student’s ranking, and the remainder used a wide variety or more or less specific numerical descriptors (e.g. quartiles, etc.).  Of those using adjectives, 60% gave complete information on the distribution of these adjectives.  Hard as it may be to believe, this represents some progress in transparency, which is sorely needed by program directors. — Laura Willett, MD

Student Perspective: Clerkship grading and evaluations are a major source of stress for medical students. Students are acutely aware of the impact clinical grades have on their MSPE and their overall residency application.  There is sometimes significant variability between clinical sites and evaluator grading even within clerkships, which can add to students’ stress.  A single evaluator with a poor understanding of the rubric can sometimes be the difference between grade levels.  I think most students would be disheartened to read about the degree of variability not only of grading systems, but also of MSPE summative performance evaluations across medical schools.  You can see why residency programs place so much emphasis on an objective measurement like USMLE Step 1 score.  It is unrealistic for medical schools to have the same clerkship grading system.  However, there should be a requirement that schools disclose the percentages of students receiving each grade per clerkship to increase transparency.  Perhaps there should also be some standardization for the summative evaluation including a standard group of adjectives and/or reporting the student’s rank in terms of quartile.  These are important discussions to have, especially as the role of Step 1 is being reconsidered. — Rebecca Krakora (4th Year Medical Student)

Westerman ME, Boe C, Bole R, Turner NS, Rose SH, Gettman MT, Thompson RH, Evaluation of Medical School Grading Variability in the United States: Are All Honors the Same? Acad Med. 2019 Jun 18.

Link To Article