- What is CREODIAS?
- Computing & Cloud
- Data & Processing
- Pricing Plans
- Examples of usage
- Example of tool usage
- Processing EO Data and Serving www services
- Processing and Storing EO
- Embedding OGC WMS Services into Your website
- GPU Use Case
- Using the EO Browser
- EO Data Finder API Manual
- Use of SNAP and QGIS on a CREODIAS Virtual Machine
- Use of WMS Configurator
- DNS as a Service - user documentation
- Use of Sinergise Sentinel Hub on the CREODIAS EO Data Hub
- Load Balancer as a Service
- Jupyter Hub
- Use of CREODIAS Finder for ordering data
- ESRI ArcGIS on CREODIAS
- Use of CEMS data through CREODIAS
- Searching, processing and analysis of Sentinel-5P data on CREODIAS
- Public Reporting Dashboards
- Sentinel Hub Documentation
- Integration Guides
- OGC API
- Custom Processing Scripts
- Legal Matters
- Partner Services
- About Us
Data & Processing
Data Related Services
EO Data HUB Service - OGC WMS extended service
Description and provisioning
These services will be provided by the so called EO Data Hub. It uses cloud infrastructure and innovative methods to efficiently process and distribute data in a matter of seconds. It can be integrated into any mapping application for web application allowing for an easy-to-use and cost-effective way to exploit the data. It removes the major hassle of downloading, archiving and processing petabytes of data and simply makes the full and global archive easily available immediately via web services. Application developers can focus on added value services and end-user applications rather than having to deal with the complexity of remote sensing data.
EO Data Hub technology is designed to work with original EO data, avoiding the need for computing intensive pre-processing and additional storage for processed tiles.
Figure 1 - EO Data Hub concept
The EO Data Hub provides the following main sets of services:
Catalogue will contain information about:
- all available meta-data for individual scene and orbit
- link to quick-look images
- support for various processing levels (L1C, L2A, etc.)
- information about planned future acquisitions with a rich number of criteria (including at least satellite unit, instrument mode, polarisation, geographical area, time window)
Catalogue service supports querying based on number of criteria:
- geographical area
- bounding box (rectangle)
- geometry (e.g. for agriculture fields)
- cloud coverage
- based on scene meta-data
- if more detailed cloud data are available, also based on location
- time interval
- absolute based on "date" or "date and time" ranges defined using ISO8601 standard (from, to, from/to)
- relative time intervals ("last week", "last month", last 2 months", "from last 6 months up to last 3 months", etc.)
- advanced time intervals ("months May-August in from 1985 to 2016)
- mosaicking priority (most recent/least recent/least cloudy)
Public service interface options:
- OGC WFS with additional parameters (details available on the web-site http://www.sentinel-hub.com/apps/wms/wfs-request)
- REST API implementing parts of OData v4.0 protocol
INSPIRE and GEOSS compatibility will be ensured for relevant data-sets.
OGC API service (rendering, visualization download)
Sentinel Hub OGC API supports
- WMS for seamless integration in GIS applications, supporting 1.1, 1.2 and 1.3 version, therefore being compatible with INSPIRE requirements;
- WCS for provision of exact EO data, best used when integrated in application developers' post-processing workflows; giving developers an ability to specify exactly, what kind of data they require (resolution, reflectance, etc.);
- WFS for meta-data provision (Catalogue access)
Standard OGC service interfaces are extended to support additional EO parameters, such as maximum cloud coverage, mosaicking order, composites, etc.
Real-time processing of EO data
Sentinel Hub OGC API is optimized for on-demand on-the-fly processing of raw (unchanged) EO data. The following steps are typically performed within a standard request:
- query Catalogue for chosen AOI, time range, cloud coverage, mission, etc.
- download necessary data from on-line storage
- decompress data
- application of pre-mosaic filters (e.g. DOS-1, statistical atmospheric correction, etc.)
- creation of mosaic based on priority (e.g. most recent data on top), cloud replacement, etc.
- compositing relevant bands to chosen EO product (true color, false color, NDVI, etc.) using chosen style (greyscale, RGB, red temperature, etc.)
- application of post-mosaic filters (color balance, HDR, midtone, gamma correction)
- re-projection to chosen CRS (e.g. Popular Web Mercator, WGS 84, national CRS systems).
- output creation in chosen file type (JPG, PNG, JP2, JPX, GeoTiff, etc.)
- compression of the output for faster download, based o user settings
Note that user can choose, based on the parameters, in case some steps should be performed or not.
A typical scenario described above takes about 1-2 second for an area of interest with a size of 1000*1000 px (depending on the chosen scale this may represent either 100 sq.km up to 25.000 sq.km).
Supported datasetsOGC API services are tailored to EO data characteristics of each individual dataset. There are two parts of customization:
- Optimization of data access, based on file type and internal file structure.
- Meta-data, spectral characteristics, color balance, projections, orthorectification, etc.
OGC API currently supports the following missions:
- Landsat 5
- Landsat 7
- Landsat 8
- ENVISAT MERIS
- Planet Scope
OGC API support for various file types is described in Architecturechapter 3.4.3
Support for additional datasets will be implemented during phase-in period as well as during the operation of the service, when new missions are introduced to CREODIAS.
Low resolution previews
EO Data Hub OGC API is optimized for high-resolution data analysis to complement the lack of similar options in other tools, which normally provide just thumbnails or small previews. Its capacity to process large amounts of data in a matter of seconds is usually throttled by the performance of EO data storage - when data from several tens of scenes need to be processed at the same time, the processing time passes a comfortable "few seconds" mark. To be able to visualize such cases, e.g. global-scale views, we use extremely low-resolution previews, which are either available within product itself or are added at ingestion stage. In case low scale pre-processed previews are required, we take care that additional storage for this does not exceed 7% mark.
When low-resolution previews are available, OGC API can perform the same actions as with high-resolution data - e.g. band composites, mosaicking, etc.
Figure 2 - Low scale preview of Sentinel-2 data acquired from 2016/07/21-2016/07/26 with less than 20% cloud coverage - it takes about 5 seconds to create output
Figure 3 - Low scale preview of Sentinel-2 data acquired from 2016/07/21-2016/07/26 with less than 20% cloud coverage- NDVI composite
Scripting environment - algorithm implementation
EO Data Hub OGC API supports ad-hoc definition of script used for evaluation of the sensor data - e.g. one can implement a new vegetation index in a matter of seconds, try out classification algorithms and similar.
The code is evaluated on-the-fly, which means that immediately after definition of new EO product, one can roll it out to complete dataset (globally, complete archive).
Supported programming interface:
- all standard programming tools (Math function, if/else, switch, etc.)
- ColorMap, ColorBlend
- access to all spectral bands
- pixel-based math, object-based math
- temporal comparison
Scripts can be applied either in the Configuration utility (WMS Configurator) or using an API parameter.
Figure 4 - Code example for NDVI within Configuration utility
Figure 5 - Classification decision tree implemented in evaluation scripting language
Figure 6 - Classification decision tree as visualized on-the-fly in Sentinel Playground
More information are available on the web-site (http://sentinel-hub.com/wms-configurator#WMSevalScript)
EO Data Hub OGC API works with imagery data stored in original projection (e.g. various UTM projections for Sentinel-2 mission).
Data conversion tools
EO Data Hub OGC WCS API can export data in various formats:
Advanced technical parameters
EO Data Hub OGC API supports several additional technical parameters, which make it possible to fine-tune the output exactly to the needs, e.g.:
- upsampling/downsampling options (nearest neighbor, bicubic, bilinear)
- color correction and enhancement options (gain, offset, histogram-based color correction, gamma, DOS-1, etc.)
- various style options (greyscale, red/green, green/white, red temperature, etc.)
Figure 7 - Nearest neighbour (left) and bicubic (right) interpolation used for upsampling
Configuration utility - WMS Configurator
Due to limited support for EO profile of OGC services in most used GIS tools (QGIS, ArcGIS and similar) we have implemented our own Configuration utility, which allows users to configure OGC WMS/WCS instances (WMS, WCS) based on their needs.
Every registered user can add as many unique-named instances as he/she chooses. An instance acts as a separate WMS/WMTS/WFS/WCS service and each can be configured to provide a certain set of layers with different settings. It is therefore possible to create multiple instances each with a different set of layers fulfilling various needs. Instances may contain an arbitrary number of layers. Each layer is associated with either one of the raw sensor bands or the products (such as TRUE_COLOR) and product styles. Layers are also additionally configurable using the settings defined above, such as MAXCC, TIME, the max area limitation, etc. The instance itself also has some global settings for default values on all layers, like image quality.
Figure 8 - Configuration utility - WMS configurator
Figure 9 - Video demonstration - Sentiel Hub - WMS configurator
Time series analysis - statistical analysis API
The Statistical analysis API performs elementary statistical computations, such as mean, standard deviation, and histogram approximating the distribution of reflectance values, on remotely sensed data for a region specified in a given spatial reference system across different bands and time ranges.
A quintessential usage example would be querying the service for basic statistics and the distribution of NDVI values for a polygon representing an agricultural unit over a time range.
The service retrieves all data from the given time range that fit the prescribed criteria, computes statistics on these data, and returns the result in the form of a JSON object.
Figure 10 - Statistical API result example; on the left side simplified result for two consecutive acquisitions, on the right side extended result containing histogram data as well
Figure 11 - An example of the statistical analysis API integrated in web application
Discovery front-end - EO Browser
EO Browser is front-end web application for querying and browsing all imagery datasets from various missions.
It supports all function needed to quickly find and use imagery data:
- search by location name or GNSS coordinates;
- zoom/pan/zoom window
- search scenes across multiple data sources using time range, cloud coverage and advanced parameters
- quick overview over available dates using smart calendar option (available dates are highlighted)
- quick view over results both in tabular form and spatial overlay
- visualization of chosen product up to full resolution in various visualization options (true color, false color, vegetation indices, custom combination of bands)
- ad-hoc evaluation of script-based band math
- export of current view to image (JPG)
- export of current view to full-resolution GeoTiff
- pinning of selected views (search parameters, visualization option, location)
- comparison of various datasets using split view or opacity slider
- (for registered users) saving of selected views for future use
- (for registered users) custom layer pre-set configurations
Figure 12 - search by location
Figure 13 - False colour visualization in full resolution
Figure 14 - Comparison of the data from Landsat 5 and Sentinel-2
EO Browser is also serving as an example of the front-end interface based on OGC services and can be used by anyone for further modifications. Source code under Apache license is available on GitHub (https://github.com/sinergise/EOBrowser).
Figure 15 - Video demonstration - EO Browser
Download service - Mosaic Generator
The Mosaic Generator serves as a user interface to Sentinel Hub OGC WCS API and provides easy way to download EO products, which can be imported in GIS tools for offline use. User needs to define area of interest, configure the parameters (e.g. mission, sensor, time range, style, etc.) and download data in standard formats (GeoTiff, JP2, JPX, etc.).
It is possible to download data units smaller than a complete product - e.g. just part of the area, specific bands or band combinations. It is also possible to download mosaics created by combining several neighboring scenes.
There are two main modes of operation for EO Data Hub Services. There are basic OGC functions which are free of charge to non-registered and registered users. These are:
- WFS access to catalogue data
- WMS with true-color composite, full resolution; meant for use in end-user applications (e.g. QGIS), not for integration in web applications.
- EO Browse
- Mosaic Generator
More advanced functions are available after monthly or longer subscription:
- WMS with various configuration options
- WCS with various configuration options
- WMTS with various configuration options
The detailed price list is available in the Price List section.
Description and provisioning
Copernicus data and information includes the user products from the Sentinels missions and value-added information provided by Copernicus Services including in-situ component. This data and information are serialized to various formats, structures and databases spread across many organizations. This heterogeneous, complex and silo-like structure has negative impact on data consumption potential.
The well designed catalogue functionality is the key component of building a single entry point to all of above mentioned resources. To accomplish this we would like to use two proven solutions and state of the art Linked Open Data approach. First – CKAN - comprehensive frontend catalogue application for information discovery. Second – EO Browser – specialized catalogue application for robust satellite imagery discovery and quick georeferenced preview.
And last but not least - Linked Open Data technologies - to boost EO data value for 3rd party processes by increasing its discoverability, linking capabilities and machine readability.
The diagram below presents general logic architecture of the data catalogue for CREODIAS.
Figure 16 - Logic of the CREODIAS Data architecture
The goal of this architecture is to address the three below mentioned factors:
- Maximizing of data and information discoverability
- Ensuring machine readability
- Enabling of external linking capabilities
Linked Open Data for EO data and information
We use state of the art – Linked Open Data technologies to boost EO data value for Third Party processes by increasing its discoverability, linking capabilities and machine readability.
The Linked Data (LD/LOD) concept was outlined by Tim Berners-Lee over ten years ago. Its main principle is that data sets published on the web can be mutually interlinked with external resources and hence become more useful in machine to machine type of interactions. It can be seen as an evolution of the Web of documents (WWW) into the Web of data, where every published resource has its own unique identifier URI [https://www.w3.org/DesignIssues/LinkedData.html].
There are three important reasons for using LOD in the CREODIAS project. First, the widening metadata search capabilities implementing the SPARQL endpoint web service. Second, the possibility of creation of interlinks from external data sets to the CREODIAS EO data resources using the RDF graph data model. Third, the implementation of publicly available satellite observations ontologies to boost machine readability by using commonly agreed terminology. All these reasons increase value of CREODIAS data by making it more discoverable and interconnected with growing global Linked Open Data universe.
GeoDCAT-AP and DGGS metadata will be used for the Catalogue. Both GeoDCAT-AP and DGGS metadata will be enriched with external links to important LOD nodes: GeoNames (LOD gazetteer) and DBPedia (LOD Wikipedia) and others. This will not only connect CREODIAS with existing LOD universe but also immediately extend search capabilities.
The above mentioned state of the art technologies have potential for developing the EO data consumption market. However to protect the existing process we will maintain the Open Search metadata interface and to satisfy INSPIRE directive regulations we will also publish ISO19115 metadata via the standard OGC CS-W service.
The Catalogue will be based on CKAN open source software which is widely used for open data publications like e.g. European Data Portal, data.org.uk or danepubliczne.gov.pl. CKAN provides user friendly web interface for all activities associated with data publication and subscription. It is capable of advanced data management. All datasets are organized and described with metadata, which allows it to be easily discoverable, with the use of search phrases and customizable filters (e.g.: tags, categories, data formats). It is possible to publish one dataset in different data formats, not only as downloadable files but also as links to web service, web API or links to external WWW resources. Datasets can be stored in CKAN, along with version history and dataset statistics, which allows to monitor the interest in datasets. CKAN also provides functionalities for collaboration, community participation and providing feedback, such as comments, ratings and sharing. CKAN is highly customizable in both terms of Look&Feel and functionalities. CKAN provides very rich RESTful JSON API, which allows other applications to discover and access the datasets. It can be integrated easily with Semantic Web technologies such as RDF data model and SPARQL.
Using this mature solution minimizes risk of implementation failure as well as provides to the CREODIAS users look & feel familiar to many of them.
CKAN will give single entry point to all information and data stored in CREODIAS repositories as well as these available in the remote repositories or in a subscription mode.
Sentinel data will be catalogued on a series level which interlink to the EO Browser for more detailed searches.
Several interfaces will be provided:
Open search interface
Existing EO Cloud search interface, based on extended version of the open source RESTO software (https://github.com/jjrom/resto). It provides API-s in OpenSearch, GEOJson and ATOM standards allowing for easy data search. The interface exposes the operational metadata catalog. It is used by EO Browser as well as exposed to external users.
Standard OGC C-SW web service capable of publishing ISO19115 metadata records through the OGC standard Catalogue Service for Web interface.
SPARQL endpoint web service implementing core of W3C specifications: SPARQL 1.1 Query Language, SPARQL 1.1 Federated Query, SPARQL 1.1 Protocol. SPARQL endpoint exposes RDF metadata as Linked Open Data from DCAT Catalogue as well as from Discrete Global Grid System RDF catalog.
Linked Data URI dereferencing
According to W3C recommendation URIs should be dereferenceable. This means that the system should respond to incoming HTTP requests with information about the resources identified by the URI. To meet this requirement we propose to dereference GeoDCAT metadata with physical archives (or proxies containing downloadable scenes, and Discrete Global Grid System metadata to Sentinel HUB serving image data for requested cell.
EO Browser – is an advanced, specialized catalogue application allowing user not only to search but also display data in a georeferenced mode. The applications will be extended with the SPARQL client functionality which will allow user to perform advanced queries using external Linked Data resources.
Sample use cases for Linked Open Data related capabilities
It’s possible to enrich the EO Cloud search engine or alternatively build external EO data explorer utilizing all spatial features stored in LinkedGeoData (LGD). LGD uses the information collected by the OpenStreetMap project and makes it available as an SPARQL endpoint according to the Linked Data principles. LGD is linked to LOD universe via GeoNames and DBPedia nodes. At the moment it stores more than 20 billion triples.
Sample search: find EO data for certain zip-codes, airport codes, points of Interests (POI), addresses, etc.
It’s possible to streamline research tasks by combining knowledge stored in Wikipedia (published in LOD as DBPedia) with CREODIAS LOD catalogue to retrieve necessary EO data in a single step.
Sample search: find most current EO data for hurricanes sites in US between 1990 and 2017 of a category 4 or above in Saffir–Simpson scale.
It’s possible to build an application which could automatically retrieve imagery for sites where severe weather condition may require action of emergency response services. The app could combine data from live RDF weather feed with EO LOD catalogue and automatically access EO data for area/s of interest.
Sample search: find most current non-cloudy S-2 image and most current S-1 imagery for area of wind 80 knots and above.
Image Intelligence support
It’s possible to build an application which could support image Intelligence analysts by automatic retrieval of imagery for the areas mentioned in the news in a certain context (e.g. terrorist attack, natural disaster, troops movements etc.). The app could use RDF news feed available through SPARQL in combination with GeoNames and EO LOD.
External indices database
It’s possible to build an external knowledge base storing historical values of various indices calculated for reference grid cells. The app can calculate value of certain index (e.g. NDVI, NDWI, etc.) and store information in the database. Using EO LOD URI allows the linking of the index value with the grid cell as well as source imagery. The database could help to study annual cycles of indices values and its variations.
Research work enrichment
It’s possible to use same concept as described above in research works to use imagery URI’s instead of actual photo to save disk space and give access to a broader picture than just a small window.
Catalogue Services are free of charge.