Toward the Routine Analysis of Diverse Data Types

This article describes a variety of data analysis problems. The types of data across these problems included free text, parallel text, an image collection, remote sensing imagery, and network packets. A strategy for approaching the analysis of these diverse types of data is described. A key part of the challenge is mapping the analytic results back into the original domain and data setting. Additionally, a common computational bottleneck encountered in each of these problems is diagnosed as analysis tools and algorithms with unbounded memory characteristics. This experience and the analysis suggest a research and development path that could greatly extend the scale of problems that can be addressed with routine data analysis tools. In particular, there are opportunities associated with developing theory and functioning algorithms with favorable memory-usage characteristics, and there are opportunities associated with developing methods and theory for describing the outcomes of analyses for the various types of data.