SYSTEM AND METHOD FOR DATA ORGANIZATION, OPTIMIZATION AND ANALYTICS

A system and method for data organization, optimization and analytics includes a web server, thrift server, distributed processing framework, key value store, distributed file system, and relational database. The web server provides a method whereby users issue control actions and query for records via interaction with the thrift server. The thrift server is the center of coordination and communication for the system and interacts with other system elements. The key value store organizes all of the operational data for the system. The key value store runs on a highly scalable distributed system, including a distributed file system for storage of data on disk. The distributed processing framework enables data to be processed in bulk and is used to execute analytical processing on the data. The relational database holds all of the administrative data in the system. Search queries are submitted by end user and results of the search query are sent from the web server to the end user. The web server sends control actions to queue background map reduce jobs. These jobs run in the distributed processing framework and are used to write data and indexes and execute bulk analytics against the key value store.