The primary goal of this project is to develop visual analytics methods and tools for high-volume sensor data streams.
The project team has worked towards three objectives (1) An end-to-end scalable visual analytic framework was developed to support high volume sensor data processing, analysis and visualization, (2) Distributed sensor data stream processing and analysis techniques were developed and implemented on the proposed system, and (3) A Novel 3D visual exploration interface were developed using VR Methods and consume-level devices to improve interactivity with the system.
The distributed high volume data stream processing and analytic techniques were demonstrated with levee surveillance dataset. A real-time water level forecasting algorithm was developed, that uses data streams from a network of sensors using a vectorized time series model. The distributed stream processing and analytics system was implemented using several tools. Storm, a distributed in-memory stream processing from TwitterTM was used to process the data streams, R was used for data pre-processing and analytics. The data is stored in MySQL, and visualized on a browser using Tableau. The dataset is visualized in 3D environment using DLP TV, camera-based motion tracker and iPod touch interface. Techniques were developed for iPod navigation of scenes displayed on TV.
The vectorized time series model outperforms other time series prediction techniques used for forecasting water levels on rivers, when the sensors have strong spatiotemporal dependency. The distributed data stream processing techniques using Storm can process 15000 streams per second on a single node versus 400 streams per second using MySQL. The prototype visualization and exploration interface was developed and demonstrated for the iLevee dataset that significantly increases the interactivity for the user as compared to visualizing the data on a browser based 2D environment.
The high volume data processing and analytics methods, and visualization techniques investigated for this project demonstrate great potential to develop next generation real-time business intelligence solutions. The developed techniques and system need several enhancements to improve the overall performance and usability of the system.