CloudFerro and its Partners are building Copernicus Data Access Service
We are proud to inform you that CloudFerro together with its partners has been selected by ESA as an industrial team to implement and operate the Copernicus Data Access initiative. The new initiative was officially kicked off. Its goal is to provide full spatial and temporal Copernicus EO data coverage, growing to over 85 PB, immediately available online for different types of users: public institutions, researchers, businesses, and individuals. The consortium is led by T-Systems and apart from CloudFerro also includes Sinergise, VITO, ACRI-ST, DLR, and RHEA. Click on their names to check their messages as well.
On Friday, 16th of December 2022, the new Copernicus Data Access Service kicked off, taking the European Union Copernicus program to the next level and ensuring the data make the greatest possible impact on institutional users, the research community, the commercial sector as well as to every citizen of our planet. The new Copernicus Data Access Service builds on existing distribution services and DIAS-es, ensuring their continuity, but it is also much more - it is bringing significant improvements. Read on to find out what is waiting for you.
Evolution of Copernicus data access
The European Union Copernicus program is the biggest provider of Earth Observation data in the world and its launch was almost certainly one of the most important events in the remote sensing community over the past decade. With systematic monitoring over large areas, good quality of data, resolution fitting its objectives, ensured the longevity of the program, and, most importantly, a clear and simple open data policy it took the world by storm. This, however, is not something to be taken for granted. When Copernicus started, remote sensing was a niche field occupying mostly researchers and the intelligence community. There was no specific reason to expect that it will be any different with Sentinel data. But it happened.
The data was picked by enthusiasts, then by companies and institutions. Fast forward a couple of years and suddenly there are hundreds of applications helping farmers to better manage their fields, financial markets forecast prices of corn, journalists regularly use it to observe or validate news, the European Common Agriculture Policy relies on its data to monitor practically all agriculture fields for sustainable agriculture practices, researchers are building digital twins of the environment, security organizations predict migration patterns... There are many more examples.
Nobody expected Copernicus data to start such a revolution in this field. It was (and still is) a tremendous validation of ESA's operational capacity, but at the same time also significant pressure on the data distribution systems as they were not expecting such uptake. Still, the Copernicus Data Hubs have already distributed an order of magnitude more data than is published. With user uptake growing further, as well as the recognized importance of Earth Observation data for the monitoring of climate change, the EU decided to invest in the next level of user data processing and distribution infrastructure. The new Copernicus Data Access Service initiative was born.
The future of European user data processing and distribution will be powered by experienced players - T-Systems and CloudFerro with their well-used cloud infrastructure, Sinergise and VITO with Sentinel Hub and OpenEO data discovery and processing tools, and DLR, ACRI-ST, and RHEA taking care of on-demand processing and Copernicus Contributing Missions. In particular CloudFerro responsibility covers the delivery of Data Space Platform and Data Retrieval providing access to the full archive of Copernicus EO data, Traceability, Security, and Cloud services for EO data processing. All of the above-mentioned services are well-versed in ESA-lead projects. Furthermore, the majority of the technology, which the service will be built upon, has been developed and validated within past ESA and Copernicus activities. The partners will work under the close guidance of ESA in order to address objectives set by the European Commission.
So, what is it all about?
First and foremost, it's about data, or rather, data availability. Instant data availability. Just about all the data ever acquired by Sentinel satellites, with minor exceptions, will be available online, instantaneously including e.g. Sentinel-1 SLC and GRD and L2 OCN, Sentinel-2 L1C and L2A (reprocessed to Collection 1, as reprocessed data becomes gradually available), Sentinel-3 and Sentinel-5P L1 and L2 data as well as Copernicus Contributing Missions data. Full archive, and always up-to-date.
There is currently no infrastructure where all of this would be available in one place. The new Copernicus Data Access will be it. Data will be available through various interfaces – from old-fashioned download, direct S3 access with STAC items, and cloud-optimized formats to streamlined access APIs, which are able not just to fetch the data but also to process it. There will be a web-based application built on top of a very popular EO Browser technology, as well as JupyterLab. And several on-demand processors are capable of building non-default formats and derived products such as Sentinel-1 coherence and CARD4L products, MAJA-powered atmospheric processing, and similar. For those interested in processing, there will be scalable cloud resources available, optimized for EO tasks. Last but not least, there will be a special focus put on traceability. With the nowadays advances in machine learning-powered image manipulation, this is especially important. For all data managed within the Copernicus Data Access, it will be possible to trace where it came from and what operations were performed on it.
Very important – the vast majority of these capabilities will be available free for use, funded by the European Union. The quotas designated for users should more than suffice for the individual's use - personal, research, or commercial. For those interested in larger scale operation, there will be practically unlimited resources available under commercial terms, powered by the industrial team partners and accessing the identical full set of data collections as well as additional open and commercial ones, allowing to build applications on a world scale. Furthermore, there will be significant credits made available, in the form of extra free resources, to be used for research and pre-commercial exploitation opportunities.
What does it mean to CREODIAS users?
This is CloudFerro's long-term commitment to grow and maintain the Copernicus EO data repository available in CREODIAS. Within the next few years, one of the largest online repositories worldwide will grow from 35PB to over 85 PB of free and immediately available EO data. Data from all over the world is available directly from the cloud through efficient, cloud-native access mechanisms (both API and GUI). Data is available for current users, new users, and federated platforms.
In addition to that, users will be able to add and disseminate their collections and processors. What is more, there will be a set of additional services available for the CREODIAS users, such as streamlined access to data, and on-demand processing.
It is all about timing, responsibilities, and commitments
The initiative has quite an intense phase-in plan, in order to allow hundreds of thousands of existing users of the Copernicus Data Hubs to migrate their workflows to the new service. First, a limited but stable, roll-out is planned for the end of January 2023 with continuous upgrades over the upcoming months until full service is made available by the end of June 2023.
This is a very challenging yet super exciting opportunity for everyone in the remote sensing community - from beginners to experts, from researchers, companies, and institutions, as well as individuals. Just about anyone can benefit from being aware of what is happening with our planet. And we should all take interest in it, and act. For the consortium partners, however, this is especially important - we were given an opportunity to build a new ecosystem for everyone to use. An ecosystem that strives to provide a significant upgrade over existing tools and services. This comes with huge responsibility towards everyone - not just to the ESA and European Commission, but for the worldwide EO community.
We are confident we can execute it, so stay tuned for further information.