Extreme Continuous Experimentation in Software Engineering

In software development, collecting data on how a software application is used to understand how the users benefit from using the application. For example, by giving two user groups different versions of the software, it is possible analyse which one provides higher value and select that into the product. The number of experiments one might want to run has increased rapidly and will continue to do so, as some companies already run running hundreds of experiments simultaneously or tens of thousands yearly.

Our approach, Extreme Continuous Experimentation, aims for running very high numbers of experiments. We aim for theoretically rigorous results that will also lay the basis for automating the experiment generation. That is, to run new experiments without constant human involvement. Automation opens new avenues for creative experiment generators that can explore feature configurations unforeseen by human developers.

Funding: Academy of Finland