Intelligence collection for counter terrorism in massive information content

During the Cold War, intelligence collection operations were primarily focused upon finding the "gold nugget" or "smoking gun". In other words, uncovering some key piece of information that could provide leads to break open an investigation. However, in the current asymmetric and counter terrorism environment, there are fewer of these gold nuggets to be found as foreign threats change from national to transnational and evidence of activity becomes harder to detect. Further, tactics used by the threat today are highly adaptive and are often well endowed in deception techniques designed to disguise plans and intent. The uncertainty of this situation presents an enormous challenge for the intelligence community, often requiring that more collection assets be deployed to ensure that adequate data are captured, which can potentially be related to some "future value". Furthermore, this "future value" can only be derived as temporally disconnected facts accumulate and become understood within a seemingly unbounded set of potential threat scenarios. Since humans are often required to translate and decipher these ephemeral facts, there is a considerable human overload issue with regard to the ratio of data ingest to effective analysis rate. In fact, the current situation in the intelligence community is one of not being able to process all of the information that is potentially available. We describe the current situation in the IC by identifying two related problems: l) The inability to identify and retain all potentially useful data that is collected, due to a lack of understanding of future utility, and the need to reduce the overall massive volume of the data; 2) The inability to find relevant data when viewed in a future context that was not understood when initially collected. This paper presents an approach for potentially dealing with these problems. The approach is based upon never discarding data, categorizing all data upon ingest, and organizing the data by sets of concepts associated with scenarios commonly encountered in counter terrorism intelligence collection. We view this approach to be somewhat analogous to a "data librarian" whose responsibility it is to catalog and store data by reference, allowing it to be applied to any variety of future contexts and scenarios. We discuss the use of mathematical transformations of data and concepts into a common pattern space where more efficient matching and categorization can be performed. Finally, we show a