Proceduralizing control and discretion: Human oversight in artificial intelligence policy

This article is an examination of human oversight in EU policy for controlling algorithmic systems in automated legal decision making. Despite the shortcomings of human control over complex technical systems, human oversight is advocated as a solution against the risks of increasing reliance on algorithmic tools. For law, human oversight provides an attractive, easily implementable and observable procedural safeguard. However, without awareness of its inherent limitations, human oversight is in danger of becoming a value in itself, an empty procedural shell used as a stand-in justification for algorithmization but failing to provide protection for fundamental rights. By complementing socio-legal analysis with Science and Technology Studies, critical algorithm studies, organization studies and human-computer interaction research, the author explores the importance of keeping the human in the loop and asks what the human element at the core of legal decision making is. Through algorithmization it is made visible how law conceptualises decision making through human actors, personalises legal decision making through the decision-maker’s discretionary power that provides proportionality and common sense, prevents gross miscarriages of justice and establishes the human encounter deemed essential for the feeling of being heard. The analysis demonstrates the necessary human element embedded in legal decision making, against which the meaningfulness of human oversight needs to be examined.