Can Students Review Their Peers?: Comparison of Peer and Instructor Reviews

Having students peer review each other's exercises is a common task in modern computing classrooms. In large classes, peer review might even partly replace traditional instructor-led review -- and prior work has found some indications that the quality of peer reviews can be close to that of instructor reviews. In this work, we explore the difference between instructor and peer reviews of student-created programming exercises. One task in an introductory programming course was to have students design their own programming exercises -- including an exercise description, model solution, and test cases -- which were then reviewed by peers. After the course, we had two instructors review the same student-created exercises. We compare the scores given by the instructors and the students to analyze potential differences. Our results suggest that agreement between instructors and students as measured by inter-rater reliability is low, although differences between instructor and student review score distributions are not statistically significant. Additionally, instructors have more fluctuation in their reviews compared to students. Due to the rising popularity of peer reviews, more research is needed to examine to what extent they could complement traditional instructor-led review of exercises.