SARF: Sparsity-Aware Reconstruction Framework for Large-Scale Datasets

Large-scale datasets, particularly those collected from smart devices and Internet of Things sensors, usually exhibit significant temporal and spatial sparsity, resulting in high amounts of missing data. Unless addressed in the analysis, this sparsity can result in substantial gaps and biases as well as limit the generalizability of conclusions drawn from such data. To address this challenge in data quality, we contribute the Sparsity-Aware Reconstruction Framework (SARF) as a novel and unified data fusion and reconstruction framework that enhances data quality and addresses sparsity. SARF analyzes datasets, partitioning the data into segments with similar characteristics, and reconstructs the data in each segment individually by selecting a reconstruction technique that is tailored to the internal temporal-spatial characteristics of the dataset. Through extensive experiments on two representative datasets - mobile application measurements and IoT sensor data from low-cost air quality sensors - we demonstrate that the targeted adaptation of reconstruction strategies employed by SARF significantly enhances the quality of reconstructed data. Our results show the robustness of SARF's performance across spatiotemporal variations, outperforming current state-of-the-art methods by margins up to 68% on average (74% for compressive sensing, 53% for convolutional sparse coding, 78% for deep learning). These findings underscore SARF's potential to enhance datadriven insights across multiple domains, paving the way for more robust analyses of sparsity-affected datasets.