Integration of IoT Streaming Data with Efficient Indexing and Storage Optimization

In the era of IoT, the world of connected experiences is created by the convergence of multiple technologies including real-time analytics, machine learning, and commodity sensors and embedded systems. However, with the proliferation of these IoT technologies and devices, there are challenges in integrating, indexing and managing time-series data from multiple sources to optimize the storage of those data and/or retrieve the information from them in real-time. Many researchers have addressed the data integration issue through developing time-series data compression techniques; however, they focused mainly on the application of integer value compression to IoT data. Moreover, existing work does not focus on the issues of data and information retrieval without decompression. In this paper, we solve these issues by constructing an indexing framework within a lossless compression for floating point time-series data, where an index is based on the time-stamp from the compressed data that facilitates the search for data without full decompression. We conduct several sets of experiments and quantify the performance of our proposed approach. The experimental results, performed on IoT datasets, show a reduction in storage compared with existing compression techniques. The experimental study also demonstrates the capability of time-series data indexing and integration in real-time.