Bringing together an ocean of information: An extensible data integration framework for biological oceanography

As increasing volumes and varieties of data are becoming available online, the challenges of accessing and using heterogeneous data resources are growing. We have developed a mediator-based data integration system called Cartel for biological oceanography data. A mediation approach is appropriate in cases where a single central warehouse is not desirable, such as when the needed data sources change frequently through time, or when there are advantages for holding heterogeneous data in their native formats. Through Cartel, data sources of a variety of types can be registered to the system, and users can query against simplified virtual schemas, without needing to know the underlying schema and computational capabilities of each data source. The system can operate on a variety of relational and geospatial data formats, and can perform joins between formats. We tested the performance of the Cartel mediator in two biological oceanography application areas, and found that the system was able to support the variety of data types needed in a typical ecology study, but that the response times were unacceptably slow when very large databases (i.e. Ocean Biogeographic Information System and the World Ocean Atlas) were used. Indexing and caching are currently being added to the system to improve response times. The mediator is an open-source product, and was developed to be a generic, extensible component available to projects developing oceanography data systems.