Cloud Cover: Two Big Data Initiatives with Ocean Applications
Ocean News
If you have attended any major ocean conferences during the past few years, you have heard the phrase: Big Data. In general, this refers to the unprecedented availability of open data that can be used to advance science, fine-tuning compute models and predictions, and improve products and services.
While a lot of catch phrases are thrown around as the next big thing, Big Data lives up to the hype. Like the invention of the steam locomotive, vaccines, and broadcasting, Big Data is revolutionizing how we live, work, and play.
Perhaps the biggest misconception among laymen is that this is nothing like what we’ve seen from the first few decades of widespread internet usage. The Big Data revolution involves much more than digitizing existing data. It involves rendering more data from our ongoing activities, often in real-time. In fact, according to the Massachusetts Institute of Technology (MIT) Media Lab, more ocean data has been collected in the last two years than in all previous years combined, and it shows no signs of slowing down.
Because there is far more available data than ever before, it can be difficult to process using traditional databases and software techniques, which is where cloud computing, cloud hosting, and cloud data storage and deployment comes into play. Cloud computing refers to the on-demand availability of computing resources delivered as a service over network (such as the Internet). In short, it’s a virtual framework. You only pay for what you use, but your ability to share data across an expandable and optimized infrastructure is increased exponentially.
NOAA’s Big Data Program
The National Oceanic and Atmospheric Administration (NOAA) generates tens of terabytes of data a day from satellites, radars, ships, weather models, and other sources. While these data are available to the public, it can be difficult to download and work with such large data volumes.
NOAA’s vast wealth of data therefore represents a substantial untapped economic opportunity. The NOAA Big Data Program (BDP) will provide public access to NOAA's open data on commercial cloud platforms through public-private partnerships. These partnerships will remove obstacles to the public use of NOAA data, help avoid costs and risks associated with federal data access services, and leverage operational public-private partnerships with the cloud computing and information services industries.
The Big Data Program combines three incredibly powerful resources: NOAA's expansive collection of high-quality environmental data and expertise, the vast infrastructure and scalable computing capabilities of our industry partners, and the innovative energy of the American economy. The BDP currently works with three infrastructure-as-a-service (IaaS) providers to broaden access to NOAA’s data resources. These partnerships are designed to not only facilitate full and open data access at no net cost to the taxpayer, but also foster innovation by bringing together the tools necessary to make NOAA's data more readily accessible.
Speaking of Accessibility . . .
NOAA’s vast collection of environmental data will be more accessible than ever before through new collaborations announced with Amazon Web Services (AWS), Google Cloud, and Microsoft. Under these new agreements, commercial cloud platform providers will generate untold opportunities for scientific and economic advances by exponentially expanding, rapid and reliable, no-cost access to NOAA data for the public.
How to Participate
For companies, organizations, and individuals interested in joining with NOAA's Big Data Project, several “data alliances” have been formed. Each data alliance is anchored by a participating Infrastructure as a Service (IaaS) institution (a “Collaborator”), and represents a market ecosystem built around one or more of NOAA’s datasets. The data alliance is a flexible concept, and may consist of larger companies that represent various economic sectors, such as the weather or insurance industries, specialized small business, value-added resellers, entrepreneurs, researchers and non-profits, etc. The data alliance structure allows market forces to act on the identification, extraction, and development of NOAA public data resources, and provides a mechanism for interested parties to work together to develop new business and research opportunities. The organizations comprising the ecosystem are free to participate in multiple data alliances.
For more information, visit one of the NOAA Big Data collaborators (below), or contact the NOAA Big Data Project Director at jon.oneil@noaa.gov.
Amazon Web Services (search the term NOAA).
EU BigDataOcean Project
The main objective of the BigDataOcean project is to propose and validate maritime big data scenarios for the benefit of EU-based companies, organizations and scientists. This is achieved through a multi-segmented platform that combines data of different velocity, variety, and volume in an inter-linked, trusted, multilingual system, producing a big-data repository of high value and veracity for project participants and local communities.
The BigDataOcean project strives to capitalize on modern technological innovations, utilizing them to revolutionize the way maritime-related industries work. The maritime sector is ripe for the introduction of innovations such as the big-data-driven economy, interrelated data streams from diverse sectors and languages, and cross-technology innovations that deliver data in several different formats (such as structured and unstructured, or real-time and in batches). These innovations will enable the creation of an entirely new value-chain, which will lead to great economic, societal, and environmental impact.
The infrastructure is being tested and facilitated through four pilots, which will provide the BigDataOcean project with an amount of data in the order of terabytes. This will result in the largest maritime database up until now, providing, among other things, collaborative, data-driven intelligence. Additionally, BigDataOcean will allow participating pilots to upload both private and public resources of data, and interrelate them through public and private queries and diagrams. The BigDataOcean system backbone will be domain-agnostic and compatible with most popular established data-processing technologies, sensor types, and common systems of operation.
Objectives
The main objective of the BigDataOcean project is to propose and validate maritime big data scenarios for the benefit of EU-based companies, organizations and scientists. This is achieved through a multi-segmented platform that combines data of different velocity, variety, and volume in an inter-linked, trusted, multilingual system, producing a big-data repository of high value and veracity for project participants and local communities.
On a technical level, BigDataOcean will provide the largest repository to date containing marine data, to facilitate big data use-cases. Furthermore, it will enhance that data with semantic information and link it to external data sources, to increase its value and make it easily reusable. Difficulties in cross-sector and multi-lingual integration will become a thing of the past, and the amount of pre-processing necessary will be far less.
In the business level, BigDataOcean will contribute by enabling different stakeholders in a collaborative but flexible (i.e. between private and public resources) repository for maritime big data, BigDataOcean will provide them with the tools (i.e. applications and APIs) to gain business logic capabilities on these data; Thus allow help them to make data-driven decisions based on diverse data resources, coming from well-trusted providers; And enable new business models that will secure sustainability and transparency in organizations operations.
On a scientific level, BigDataOcean will allow scientists to easily automate hypothesis testing and analysis, as well as facilitate the extraction of data models. These processes will be further simplified via the structural enhancement that the data in the repository undergoes from different sources. Additionally, providing scientists with the repository allows them to focus on building better models and algorithms without having to worry about the data required to do so. By reducing the barriers of analyzing maritime data, this type of research is made more appealing for stakeholders that are outside academia who wish to run research tasks (e.g. determining causality between factors).
BigDataOcean Ecosystem and Use Cases
Marine areas cover almost 72% of our planet. Still, almost 95% of this realm remains unexplored, hiding many secrets and scientific phenomena. Despite its unknown and unpredictable nature, the sea has been since the ancient times one of the most valuable and exploited “economic platforms” of mankind, with applications crossing different sectors. Indeed, around 80% of the global trade volume is carried by sea, corresponding to 70% of global trade value.
BigDataOcean positions itself to contribute and improve that economic potential, using big data applications developed within four major use cases in the maritime environment:
- Pilot 1 – Vessels’ Operation Improvement. Naval engineers and shipping companies try to constantly minimize fixed and operational costs, as well as their impact on the maritime domain. As the name suggests, this pilot uses data from every available sensor in the vessel to approach maintenance aspects proactively rather than reactively. It also analyses the impact of the environmental conditions and the operational decisions on the vessel’s fuel consumption.
- Pilot 2 – Mare Protection. This pilot provides a series of scenarios based on the POSEIDON Oil Spill Model, offering essential information for the effective response and management of possible oil spill pollution incidents. Forecasting simulations, enhanced with various cross-sectoral marine data, are taking advantage of the variety and extended ocean data volume available in the BigDataOcean platform.
- Pilot 3 – Maritime Security and Anomaly Detection. Events, activities and threats in the maritime ecosystem can potentially impact global safety, economic activity or the environment. Understanding of such events, activities, and threats, known as Maritime Situational Awareness (MSA), has been ineffective in the past due to the lack of data. With BigDataOcean that is changing and is now possible to identify vessel routes based on their motion patterns, enabling proactive measures to minimise threats at sea.
- Pilot 4 – Wave Power as the Next Clean Energy Source. This pilot is exploiting the relationship between two major domains, i.e. the maritime and the energy production, and can clearly contribute to a great exploration of the marine areas. The services available use use multiple numerical models and data sources to evaluate the wave energy potential in diverse regions, hence contributing to the future development of wave energy solutions.
How to Participate
While some of the use cases are of a private nature, Pilots 2, 3, and 4 are available upon registration for testing in the BigDataOcean Platform.
