The use of artificial intelligence (AI) programs such as ChatGPT is a source of controversy in academia, including in the college admissions process. Are applicants for dermatology residency programs using AI to write or enhance their personal statements, and if so, to what extent? A study published in the May Journal of Drugs in Dermatology found a significant increase in AI usage in dermatology residency application personal statements from 2022 to 2024.
I interviewed author Priya Patel Housley, MS, of the University of Florida College of Medicine.
What led you to want to investigate AI usage in dermatology residency application personal statements?
In late 2022, large language models (LLMs) were becoming freely available with user-friendly interfaces. The most recent application cycle (2023), at the time of the study, was the first in which applicants had wide access to these tools. Due to the nature of personal statements, which are typically in an applicant’s voice, we were interested to see if this technology may now be encroaching on a truly personal component of the application. As programs place varying weights on the importance of the statement, those that highly consider them may experience an impact in the authenticity of their resident selection process.
You analyzed deidentified personal statements and found a significant increase in AI-generated content in 2024 compared with previous years. How did you conduct your analysis?
We collected 1500 dermatology personal statements evenly split between the three application cycles 2022, 2023, and 2024. For our positive controls: we generated essays with the ChatGPT models available at the time. We also collected human written control personal statements that were submitted before 2022 and used them as our negative controls. These three groups of essays were processed through AI detection software and the output percentage of AI detection was used to do a series of statistical testing shown below.
Two of the AI detection software programs you used found a significant increase in the percentage of essays in which AI generated text was detected between 2022 and 2024. Were you surprised by the sharp increase?
Although we were not surprised by the increase, the rate in which students adopted the usage of LLMs and trusted the tool in such a career defining application is striking. For example, if applicants used ChatGPT, there was a window of only several months from public availability to residency application submission and fewer months assuming that applicants prepared their personal statements ahead of the September deadline. It seems applicants were quick to trust and accept the technology, and I predict that these values will only increase unless usage guidelines are introduced.
Why do you think so many dermatology residency applicants sought at least partial help from AI with their personal statements?
The transition between pre-clinical and clerkship years is notoriously known for being challenging and time intensive. In addition, many programs stress the importance of clerkship grades and medical licensing exam (USMLE Step 2) performance, especially for competitive specialties such as dermatology. As a result, applicants may seek tools that lessen their load. LLMs are especially appealing due to their ease of use and human-like content generation. In addition, applicants may be concerned that the bar is being raised by those that do use these tools and may feel the need to match the quality of their peers’ personal statements. Both applicants and program directors should be aware, however, that many LLMs leave watermarks, which are traceable identifiers in the text, which is also the basis of the detection software we used.
Why are the results of your study important for residency program directors to consider?
As previous metrics utilized by program directors have become binary (pass/fail pre-clinical grades, Step 1 medical licensing exam scores), emphasis has been placed on performing holistic application reviews, entailing a possible greater weight for personal statements. With increasing detection of LLM usage, program directors may want to consider how they utilize personal statements in their interview selection process. Furthermore, program directors may want to consider how they regulate AI usage in application materials submitted to their programs. Would they like to encourage disclosure on usage, prohibit usage, or allow technology to take its course? Would they allow LLMs for editing purposes but not content generation? Would program directors want the residency application service to implement these policies and possibly provide an AI detection report? At the very least, program directors should be aware that LLM usage is on the rise. While some statements may no longer reflect an applicant’s own voice, others will continue to be genuine – a challenging distinction to make, as to not penalize the latter group of applicants.
Did you enjoy this JDD Buzz article? You can find more here.
