A Prompt for Generating Script Concordance Test UsingChatGPT, Claude, and Llama Large Language ModelChatbots

Abstract

Medical education always evolves to incorporate more tools for specific needs in assessing clinicalreasoning skills. Among these tools, Script Concordance Test (SCT) has a particular importancedue to its focus on assessing decision-making in uncertain clinical situations. However,development of SCT items is effortful. Artificial intelligence tools, such as large language models,offer significant benefits. These models are already used for generating multiple-choice questions,and their use in generating SCTs offers great promise. However, this requires well-designedprompts to generate SCTs. This article proposes a generic prompt for the ChatGPT-4, Claude 3,Llama 3, and ChatGPT-4o large language model chatbots to generate SCTs, which can be tailoredto various fields of medicine and different stages of medical education. It can help to streamlinethe development process of SCTs. Initial findings are promising, and there is a need for generatingSCTs using large language models and conducting research to assess the quality of SCTs