AI See What You Did There – The Prevalence of LLM-Generated Answers in MOOC Responses

Large language models (LLMs) are reshaping the educational landscape, particularly in online learning environments where student supervision is often limited. Early evidence and anecdotal reports suggest that the use of AI-generated content is highly prevalent among students. However, definitive statistics remain elusive, primarily due to the challenges associated with distinguishing between AI-generated and human-generated responses. Establishing clear evidence and effective mechanisms for identifying AI-generated responses is crucial for understanding the significance of this challenge and for developing policies to address it. To tackle these issues, we present a large-scale empirical study on the prevalence of AI-generated content in online education. Our study analyzes over 4045 student responses from an introductory MOOC on the Internet of Things, employing textual analysis techniques to evaluate various metrics for identifying AI-generated responses and understanding their characteristics. Our findings reveal that a significant majority of student responses (up to 90.1%) exhibit strong similarities to AI-generated content in both wording and contextual meaning, regardless of the specific LLM or similarity metric employed. In terms LLMs usage, DeepSeek, Gemini, and Grok are the three most popular LLMs used to generate responses.