Comparison of Automatic Item Generation Methods in the Assessment of Clinical Reasoning Skills

Abstract

The use of automatic item generation (AIG) methods offers potential for assessingclinical reasoning (CR) skills in medical education, a critical skill combining intuitive andanalytical thinking. In preclinical education, these skills are commonly evaluated through writtenexams and case-based multiple-choice questions (MCQs), which are widely used due to the highnumber of students, ease of standardization, and quick evaluation. This research generated CR-focused questions for medical exams using two primary AIG methods: template-based and non-template-based (using AI tools like ChatGPT for a flexible approach). A total of 18 questions wereproduced on ordering radiologic investigations for abdominal emergencies, alongside faculty-developed questions used in medical exams for comparison. Experienced radiologists evaluatedthe questions based on clarity, clinical relevance, and effectiveness in measuring CR skills. Resultsshowed that ChatGPT-generated questions measured CR skills with an 84.52% success rate,faculty-developed questions with 82.14%, and template-based questions with 78.57%, indicatingthat both AIG methods are effective in CR assessment, with ChatGPT performing slightly better.Both AIG methods received high ratings for clarity and clinical suitability, showing promise inproducing effective CR-assessing questions comparable to, and in some cases surpassing, faculty-developed questions. While template-based AIG is effective, it requires more time and effort,suggesting that both methods may offer time-saving potential in exam preparation for educators.