EODATA Related Services

PGaaS - Product Generation as a Service

Product Generation as a Service PGaaS is a serverless processing service, which allows easy mass processing of EO products without using virtual machines. The Product Generation solution allows the user to save time related to preparing the environment for the implementation of the required processors. In this scenario, the user does not need to configure the operating system and the entire cloud environment required to scale the processing to get results faster.
Our service allows the user to easily select the type of processor he wants to use. After choosing the processor, the user defines the group of products to be processed by selecting an area of interest, a date range and optionally other parameters. The user may also define additional processor output parameters if needed.

The service is available in two modes:

• best effort
• premium.

The best effort scenario assumes queuing orders on a specific pool of cloud resources, with no guarantee that the processing will be done in a specific time. In this case the customer's order is processed on limited, shared resources. In this mode, the service is free of charge. This solution is best for projects that require several dozen or several tens of processed products. In case of a large queue and another few hundred products, the waiting time for the completion of processing may take even several weeks.

The premium service uses the full potential of CloudFerro’s cloud environment, which allows users to generate thousands of products in a limited, well defined timeframe. To activate this mode, the user should contact the sales department and agree on individual processing conditions. An appropriate pool of cloud resources will be allocated, to guarantee the generation of the products in a limited timeframe. Premium services are billed according to a price list. Processing costs depend on the number of processed products and the type of processor selected.

Below there is a list of available processors:

Sentinel-1 CARD-BS
The Sentinel-1 CARD-BS processor produces Sentinel-1 Level-2 terrrain corrected backscatter data, expressed in Gamma0 units for both VH and VV polarisations. The processor uses ESA SNAP software and the SNAP workflow consists of the following steps: Applying Orbit File, Removal of Border Noise, Removal of Thermal Noise, Calibration and Terrain Correction. The processor ingests Sentinel-1 GRD data and uses automatically downloaded orbit and calibration data for the initial corrections and calibrations, as well as customly mosaicked Copernicus DEM 30m elevation data for the terrain correction. The output data is formatted in DIMAP format.

Sentinel-1 CARD-COH6
The Sentinel-1 CARD-COH6 processor produces Sentinel-1 Level-2 interferometric coherence data, based on a 6-day temporal baseline for both VH and VV polarisations. The processor uses ESA SNAP software for the coherence estimations for two image pair combinations (the primary image and its two overlapping secondary images acquired six days prior) followed by a mosaicking step performed with GDAL. The SNAP workflow consists of the following steps: Applying Orbit File, TOPSAR Split, Back-Geocoding, Coherence, TOPSAR Deburst, TOPSAR Merge, Multilook and Terrain Correction. The processor ingests Sentinel-1 SLC data and uses automatically downloaded orbit data for the initial corrections, as well as customly mosaicked Copernicus DEM 30m elevation data for the terrain correction. The final mosaicking step in GDAL merges the two coherence images in one final product covering the entire footprint of the primary product. The output data is formatted in Geotiff format, with the two bands indicating the VH and VV polarisations, respectively.

Sentinel-1 COG
The Sentinel-1 COG processor produces Sentinel-1 Level-1C GRD data in Cloud-Optimised Geotiff (COG) format, using a gdal_translate conversion routine in GDAL. The processor ingests Sentinel-1 GRD DIMAP data in the measurement directory of the .SAFE image and converts it to Geotiff format. The .SAFE file structure of the image stays the same as the original image.

Sentinel-2 sen2cor L2A
The Sentinel-2 sen2cor Level-2 processor performs atmospheric correction of Sentinel-2 Level-1C data using the sen2cor v2.8.0 processor. Parameters of the processor as the same as the settings used by Copernicus/ESA for their in-house Sentinel-2 Level-2 processor, except for the DEM data (ESA uses PlanetDEM, CREODIAS uses Copernicus DEM 90m). A small collection of old Sentinel-2 L1C images processed with an old processing baseline (from 2015 and 2016) are processed with sen2cor v2.5.5. The sen2cor version selection is performed automatically by the processor and stored in the metadata. When a user selects a Sentinel-2 L1C image, the processor initially checks whether a corresponding Sentinel-2 L2A image already exists in the CREODIAS repository, subsequently whether a corresponding Sentinel-2 L2A image already exists in the Copernicus repository (if so, it downloads it to the CREODIAS repository. Only if no Sentinel-2 L2A image exists in either repository, the Sentinel-1 L1C product will be processed to Sentinel-2 L2A with the sen2cor processor.

Sentinel-2: Resolution Enhancer
Sentine-2: Resolution Enhancer allows for object enhancement in images for 8 spectral channels with simultaneous preservation of radiometric quality. The result of the processing is a Sentinel-2 L2A product with a resolution of 2.5m GSD, saved as a multilayer geoTIFF file. The solution gives the possibility to access current and historical data of the Sentinel-2 in the so far unavailable resolution

In order to facilitate the use of premium services, we have prepared a simple and transparent price list based on the selected processor type and number of products to be generated.
Please visit our price list page for PGaaS pricing.

All available information such as tutorials on how to order processing can be found in our FAQ section at this address.
If you have any questions about PGaaS please contact our sales department.

EO Data HUB Service - OGC WMS extended service

Description and provisioning
These services will be provided by the so called EO Data Hub.  It uses cloud infrastructure and innovative methods to efficiently process and distribute data in a matter of seconds. It can be integrated into any mapping application for web application allowing for an easy-to-use and cost-effective way to exploit the data. It removes the major hassle of downloading, archiving and processing petabytes of data and simply makes the full and global archive easily available immediately via web services. Application developers can focus on added value services and end-user applications rather than having to deal with the complexity of remote sensing data.

EO Data Hub technology is designed to work with original EO data, avoiding the need for computing intensive pre-processing and additional storage for processed tiles.
 

Figure 1 - EO Data Hub concept

The EO Data Hub provides the following main sets of services:
Catalogue Service

Catalogue will contain information about:

• scenes/tiles
• orbits
• all available meta-data for individual scene and orbit
• link to quick-look images
• support for various processing levels (L1C, L2A, etc.)
• information about planned future acquisitions with a rich number of criteria (including at least satellite unit, instrument mode, polarisation, geographical area, time window)

Catalogue service supports querying based on number of criteria:

• geographical area
  • bounding box (rectangle)
  • geometry (e.g. for agriculture fields)
• mission/sensor
• cloud coverage
  • based on scene meta-data
  • if more detailed cloud data are available, also based on location
• time interval
  • absolute based on "date" or "date and time" ranges defined using ISO8601 standard (from, to, from/to)
  • relative time intervals ("last week", "last month", last 2 months", "from last 6 months up to last 3 months", etc.)
  • advanced time intervals ("months May-August in from 1985 to 2016)
• mosaicking priority (most recent/least recent/least cloudy)

Public service interface options:

• OGC WFS with additional parameters (details available on the web-site http://www.sentinel-hub.com/apps/wms/wfs-request)
• REST API implementing parts of OData v4.0 protocol  

INSPIRE and GEOSS compatibility  will be ensured for relevant data-sets.

OGC API service (rendering, visualization download)

Sentinel Hub OGC API supports

• WMS for seamless integration in GIS applications, supporting 1.1, 1.2 and 1.3 version, therefore being compatible with INSPIRE requirements;
• WMTS;
• WCS for provision of exact EO data, best used when integrated in application developers' post-processing workflows; giving developers an ability to specify exactly, what kind of data they require (resolution, reflectance, etc.);
• WFS for meta-data provision (Catalogue access)

Standard OGC service interfaces are extended to support additional EO parameters, such as maximum cloud coverage, mosaicking order, composites, etc.

Real-time processing of EO data

Sentinel Hub OGC API is optimized for on-demand on-the-fly processing of raw (unchanged) EO data. The following steps are typically performed within a standard request:

• query Catalogue for chosen AOI, time range, cloud coverage, mission, etc.
• download necessary data from on-line storage
• decompress data
• application of pre-mosaic filters (e.g. DOS-1, statistical atmospheric correction, etc.)
• creation of mosaic based on priority (e.g. most recent data on top), cloud replacement, etc.
• compositing relevant bands to chosen EO product (true color, false color, NDVI, etc.) using chosen style (greyscale, RGB, red temperature, etc.)
• application of post-mosaic filters (color balance, HDR, midtone, gamma correction)
• re-projection to chosen CRS (e.g. Popular Web Mercator, WGS 84, national CRS systems).
• output creation in chosen file type (JPG, PNG, JP2, JPX, GeoTiff, etc.)
• compression of the output for faster download, based o user settings

Note that user can choose, based on the parameters, in case some steps should be performed or not.

A typical scenario described above takes about 1-2 second for an area of interest with a size of 1000*1000 px (depending on the chosen scale this may represent either 100 sq.km up to 25.000 sq.km).

Supported datasets

• Optimization of data access, based on file type and internal file structure.
• Meta-data, spectral characteristics, color balance, projections, orthorectification, etc.

OGC API currently supports the following missions:

• Sentinel-2
• Sentinel-3
• Landsat 5
• Landsat 7
• Landsat 8
• ENVISAT MERIS
• RapidEye
• Planet Scope

OGC API support for various file types is described in Architecturechapter 3.4.3
Support for additional datasets will be implemented during phase-in period as well as during the operation of the service, when new missions are introduced to CREODIAS.

Low resolution previews

EO Data Hub OGC API is optimized for high-resolution data analysis to complement the lack of similar options in other tools, which normally provide just thumbnails or small previews. Its capacity to process large amounts of data in a matter of seconds is usually throttled by the performance of EO data storage - when data from several tens of scenes need to be processed at the same time, the processing time passes a comfortable "few seconds" mark. To be able to visualize such cases, e.g. global-scale views, we use extremely low-resolution previews, which are either available within product itself or are added at ingestion stage. In case low scale pre-processed previews are required, we take care that additional storage for this does not exceed 7% mark.

When low-resolution previews are available, OGC API can perform the same actions as with high-resolution data - e.g. band composites, mosaicking, etc.
 

Figure 2 - Low scale preview of Sentinel-2 data acquired from 2016/07/21-2016/07/26 with less than 20% cloud coverage - it takes about 5 seconds to create output

Figure 3 - Low scale preview of Sentinel-2 data acquired from 2016/07/21-2016/07/26 with less than 20% cloud coverage- NDVI composite

Scripting environment - algorithm implementation

EO Data Hub OGC API supports ad-hoc definition of script used for evaluation of the sensor data - e.g. one can implement a new vegetation index in a matter of seconds, try out classification algorithms and similar.

The code is evaluated on-the-fly, which means that immediately after definition of new EO product, one can roll it out to complete dataset (globally, complete archive).

Supported programming interface:

• JavaScript
• Python

Supported functions

• all standard programming tools (Math function, if/else, switch, etc.)
• ColorMap, ColorBlend
• access to all spectral bands
• pixel-based math, object-based math
• temporal comparison

Scripts can be applied either in the Configuration utility (WMS Configurator) or using an API parameter.

Figure 4 - Code example for NDVI within Configuration utility

Figure 5 - Classification decision tree implemented in evaluation scripting language

 
 
Figure 6 - Classification decision tree as visualized on-the-fly in Sentinel Playground

More information are available on the web-site (http://sentinel-hub.com/wms-configurator#WMSevalScript)

Reprojection

EO Data Hub OGC API works with imagery data stored in original projection (e.g. various UTM projections for Sentinel-2 mission).

Data conversion tools

EO Data Hub OGC WCS API can export data in various formats:

• GeoTIFF
• JP2
• JPX
• PNG
• JPG

Advanced technical parameters

EO Data Hub OGC API supports several additional technical parameters, which make it possible to fine-tune the output exactly to the needs, e.g.:

• upsampling/downsampling options (nearest neighbor, bicubic, bilinear)
• color correction and enhancement options (gain, offset, histogram-based color correction, gamma, DOS-1, etc.)
• various style options (greyscale, red/green, green/white, red temperature, etc.)
   

Figure 7 - Nearest neighbour (left) and bicubic (right) interpolation used for upsampling

Configuration utility - WMS Configurator

Due to limited support for EO profile of OGC services in most used GIS tools (QGIS, ArcGIS and similar) we have implemented our own Configuration utility, which allows users to configure OGC WMS/WCS instances (WMS, WCS) based on their needs.

Every registered user can add as many unique-named instances as he/she chooses. An instance acts as a separate WMS/WMTS/WFS/WCS service and each can be configured to provide a certain set of layers with different settings. It is therefore possible to create multiple instances each with a different set of layers fulfilling various needs. Instances may contain an arbitrary number of layers. Each layer is associated with either one of the raw sensor bands or the products (such as TRUE_COLOR) and product styles. Layers are also additionally configurable using the settings defined above, such as MAXCC, TIME, the max area limitation, etc. The instance itself also has some global settings for default values on all layers, like image quality.

Figure 8 - Configuration utility - WMS configurator

Figure 9 - Video demonstration - Sentiel Hub - WMS configurator

Time series analysis - statistical analysis API

The Statistical analysis API performs elementary statistical computations, such as mean, standard deviation, and histogram approximating the distribution of reflectance values, on remotely sensed data for a region specified in a given spatial reference system across different bands and time ranges.

A quintessential usage example would be querying the service for basic statistics and the distribution of NDVI values for a polygon representing an agricultural unit over a time range.

The service retrieves all data from the given time range that fit the prescribed criteria, computes statistics on these data, and returns the result in the form of a JSON object.

Figure 10 - Statistical API result example; on the left side simplified result for two consecutive acquisitions, on the right side extended result containing histogram data as well

Figure 11 - An example of the statistical analysis API integrated in web application

Discovery front-end - EO Browser

EO Browser is front-end web application for querying and browsing all imagery datasets from various missions.

It supports all function needed to quickly find and use imagery data:

• search by location name or GNSS coordinates;
• zoom/pan/zoom window
• search scenes across multiple data sources using time range, cloud coverage and advanced  parameters
• quick overview over available dates using smart calendar option (available dates are highlighted)
• quick view over results both in tabular form and spatial overlay
• visualization of chosen product up to full resolution in various visualization options (true color, false color, vegetation indices, custom combination of bands)
• ad-hoc evaluation of script-based band math
• export of current view to image (JPG)
• export of current view to full-resolution GeoTiff
• pinning of selected views (search parameters, visualization option, location)
• comparison of various datasets using split view or opacity slider
• (for registered users) saving of selected views for future use
• (for registered users) custom layer pre-set configurations

Figure 12 - search by location

Figure 13 - False colour visualization in full resolution

Figure 14 - Comparison of the data from Landsat 5 and Sentinel-2

EO Browser is also serving as an example of the front-end interface based on OGC services and can be used by anyone for further modifications. Source code under Apache license is available on GitHub (https://github.com/sinergise/EOBrowser).

Figure 15 - Video demonstration - EO Browser

Download service - Mosaic Generator

The Mosaic Generator serves as a user interface to Sentinel Hub OGC WCS API and provides easy way to download EO products, which can be imported in GIS tools for offline use. User needs to define area of interest, configure the parameters (e.g. mission, sensor, time range, style, etc.) and download data in standard formats (GeoTiff, JP2, JPX, etc.).

It is possible to download data units smaller than a complete product - e.g. just part of the area, specific bands or band combinations. It is also possible to download mosaics created by combining several neighboring scenes.

Billing
There are two main modes of operation for EO Data Hub Services. There are basic OGC functions which are free of charge to non-registered and registered users. These are:

• WFS access to catalogue data
• WMS with true-color composite, full resolution; meant for use in end-user applications (e.g. QGIS), not for integration in web applications.
• EO Browse
• Mosaic Generator

More advanced functions are available after monthly or longer subscription:

• WMS with various configuration options
• WCS with various configuration options
• WMTS with various configuration options

The detailed price list is available in the Price List section.

Catalogue Services

Description and provisioning

Copernicus data and information includes the user products from the Sentinels missions and value-added information provided by Copernicus Services including in-situ component. This data and information are serialized to various formats, structures and databases spread across many organizations. This heterogeneous, complex and silo-like structure has negative impact on data consumption potential.

The well designed catalogue functionality is the key component of building a single entry point to all of above mentioned resources. To accomplish this we would like to use two proven solutions and state of the art Linked Open Data approach. First – CKAN - comprehensive frontend catalogue application for information discovery. Second – EO Browser – specialized catalogue application for robust satellite imagery discovery and quick georeferenced preview.   

And last but not least - Linked Open Data technologies - to boost EO data value for 3rd party processes by increasing its discoverability, linking capabilities and machine readability.

The diagram below presents general logic architecture of the data catalogue for CREODIAS.

Figure 16 - Logic of the CREODIAS Data architecture

The goal of this architecture is to address the three below mentioned factors:

• Maximizing of data and information discoverability
• Ensuring machine readability
• Enabling of external linking capabilities

Linked Open Data for EO data and information

We use state of the art – Linked Open Data technologies to boost EO data value for Third Party processes by increasing its discoverability, linking capabilities and machine readability.

The Linked Data (LD/LOD) concept was outlined by Tim Berners-Lee over ten years ago. Its main principle is that data sets published on the web can be mutually interlinked with external resources and hence become more useful in machine to machine type of interactions. It can be seen as an evolution of the Web of documents (WWW) into the Web of data, where every published resource has its own unique identifier URI [https://www.w3.org/DesignIssues/LinkedData.html].

There are three important reasons for using LOD in the CREODIAS project. First, the widening metadata search capabilities implementing the SPARQL endpoint web service. Second, the possibility of creation of interlinks from external data sets to the CREODIAS EO data resources using the RDF graph data model. Third, the implementation of publicly available satellite observations ontologies to boost machine readability by using commonly agreed terminology. All these reasons increase value of CREODIAS data by making it more discoverable and interconnected with growing global Linked Open Data universe.

GeoDCAT-AP and DGGS metadata will be used for the Catalogue. Both GeoDCAT-AP and DGGS metadata will be enriched with external links to important LOD nodes: GeoNames (LOD gazetteer) and DBPedia (LOD Wikipedia) and others. This will not only connect CREODIAS with existing LOD universe but also immediately extend search capabilities.

The above mentioned state of the art technologies have potential for developing the EO data consumption market. However to protect the existing process we will maintain the Open Search metadata interface and to satisfy INSPIRE directive regulations we will also publish ISO19115 metadata via the standard OGC CS-W service.

The Catalogue will be based on CKAN open source software which is widely used for open data publications like e.g. European Data Portal, data.org.uk or danepubliczne.gov.pl. CKAN provides user friendly web interface for all activities associated with data publication and subscription. It is capable of advanced data management. All datasets are organized and described with metadata, which allows it to be easily discoverable, with the use of search phrases and customizable filters (e.g.: tags, categories, data formats). It is possible to publish one dataset in different data formats, not only as downloadable files but also as links to web service, web API or links to external WWW resources. Datasets can be stored in CKAN, along with version history and dataset statistics, which allows to monitor the interest in datasets. CKAN also provides functionalities for collaboration, community participation and providing feedback, such as comments, ratings and sharing. CKAN is highly customizable in both terms of Look&Feel and functionalities. CKAN provides very rich RESTful JSON API, which allows other applications to discover and access the datasets. It can be integrated easily with Semantic Web technologies such as RDF data model and SPARQL.

Using this mature solution minimizes risk of implementation failure as well as provides to the CREODIAS users look & feel familiar to many of them.  

CKAN will give single entry point to all information and data stored in CREODIAS repositories as well as these available in the remote repositories or in a subscription mode.
Sentinel data will be catalogued on a series level which interlink to the EO Browser for more detailed searches.

Several interfaces will be provided:

Open search interface

Existing EO Cloud search interface, based on extended version of the open source RESTO software (https://github.com/jjrom/resto). It provides API-s in OpenSearch, GEOJson and ATOM standards allowing for easy data search. The interface exposes the operational metadata catalog. It is used by EO Browser  as well as exposed to external users.

C-SW interface

Standard OGC C-SW web service capable of publishing ISO19115 metadata records through the OGC standard  Catalogue Service for Web interface.

SPARQL interface

SPARQL endpoint web service implementing core of W3C specifications: SPARQL 1.1 Query Language, SPARQL 1.1 Federated Query, SPARQL 1.1 Protocol. SPARQL endpoint exposes RDF metadata as Linked Open Data from DCAT Catalogue as well as from Discrete Global Grid System RDF catalog.

Linked Data URI dereferencing

According to W3C recommendation URIs should be dereferenceable. This means that the system should respond to incoming HTTP requests with information about the resources identified by the URI. To meet this requirement we propose to dereference GeoDCAT metadata with physical archives (or proxies containing downloadable scenes, and Discrete Global Grid System metadata to Sentinel HUB serving image data for requested cell.

EO Browser

EO Browser – is an advanced, specialized catalogue application allowing user not only to search but also display data in a georeferenced mode. The applications will be extended with the SPARQL client functionality which will allow user to perform advanced queries using external Linked Data resources.

Sample use cases for Linked Open Data related capabilities

Extended geo-search

It’s possible to enrich the EO Cloud search engine or alternatively build external EO data explorer utilizing all spatial features stored in LinkedGeoData (LGD). LGD uses the information collected by the OpenStreetMap project and makes it available as an SPARQL endpoint according to the Linked Data principles. LGD is linked to LOD universe via GeoNames and DBPedia nodes. At the moment it stores more than 20 billion triples.
Sample search: find EO data for certain zip-codes, airport codes, points of Interests (POI), addresses, etc.

Environmental research

It’s possible to streamline research tasks by combining knowledge stored in Wikipedia (published in LOD as DBPedia) with CREODIAS LOD catalogue to retrieve necessary EO data in a single step.
Sample search:  find most current EO data for hurricanes sites in US between 1990 and 2017 of a category 4 or above in Saffir–Simpson scale.

Emergency response

It’s possible to build an application which could automatically retrieve imagery for sites where severe weather condition may require action of emergency response services. The app could combine data from live RDF weather feed with EO LOD catalogue and automatically access EO data for area/s of interest.
Sample search: find most current non-cloudy S-2 image and most current S-1 imagery for area of wind 80 knots and above.

Image Intelligence support

It’s possible to build an application which could support image Intelligence analysts by automatic retrieval of imagery for the areas mentioned in the news in a certain context (e.g. terrorist attack, natural disaster, troops movements etc.). The app could use RDF news feed available through SPARQL in combination with GeoNames and EO LOD.

External indices database

It’s possible to build an external knowledge base storing historical values of various indices calculated for reference grid cells. The app can calculate value of certain index (e.g. NDVI, NDWI, etc.) and store information in the database. Using EO LOD URI allows the linking of the index value with the grid cell as well as source imagery. The database could help to study annual cycles of indices values and its variations.

Research work enrichment

It’s possible to use same concept as described above in research works to use imagery URI’s instead of actual photo to save disk space and give access to a broader picture than just a small window.

Billing
Catalogue Services are free of charge.