- What is CREODIAS?
- Computing & Cloud
- Data & Processing
- Pricing Plans
- Fight with COVID-19
- Examples of usage
- Example of tool usage
- Processing EO Data and Serving www services
- Processing and Storing EO
- Embedding OGC WMS Services into Your website
- GPU Use Case
- Using the EO Browser
- EO Data Finder API Manual
- Use of SNAP and QGIS on a CREODIAS Virtual Machine
- Use of WMS Configurator
- DNS as a Service - user documentation
- Use of Sinergise Sentinel Hub on the CREODIAS EO Data Hub
- Load Balancer as a Service
- Jupyter Hub
- Use of CREODIAS Finder for ordering data
- ESRI ArcGIS on CREODIAS
- Use of CEMS data through CREODIAS
- Searching, processing and analysis of Sentinel-5P data on CREODIAS
- ASAR data available on CREODIAS
- Public Reporting Dashboards
- Sentinel Hub Documentation
- Integration Guides
- OGC API
- Custom Processing Scripts
- Legal Matters
- Partner Services
- About Us
Data & Processing
DIAS Data Access Interfaces
In order to provide a very flexible functionality for Third Party Users, the CREODIAS data can be accessed using a complete set of interfaces. Raw data is stored on the Platform in its native format to ensure 100% conformity to the source data. It can be accessed directly through a standard HTTP SWIFT/S3 object interface. Legacy applications, virtual machines or interactive users can access the same data through a POSIX standard interface using filesystem emulation . Data stored in the repository can be also dynamically published in the form of user-configurable OGC WMS/WMTS/WCS web services provided by the EO Data Processing/Access HUB. Finally, CREODIAS provides a general purpose web interface for end-users – the EO Browser. Data products can also be accessed via HTTP Download while information services can be made accessible via an http Proxy service.
All CREODIAS data access interfaces are open source. The access methods are detailed in the following Sections.
The availability of the Data Access Interfaces depends on the CREODIAS Data Location. When data is present in the CREODIAS Locations Local and/or Cached – then Object Data SWIFT/S3, EO Data Hub (OGC related) and HTTP Download can be used. When data is in Location Orderable – HTTP Download can be used but this kind of data can be also Ordered to become Cached and be available as described above. The data or information in Location External can be accessed via Proxy interface.
The available Data Access Interfaces are described in general in sections below. Full details of the interfaces will be available in the Support Framework in the DIAS Web Site.
Object Data Access API (SWIFT/S3)
The object access API is the main access method for CREODIAS data.
Tenants can access EO data by sending HTTP requests to a standard SWIFT or S3 object interface. Appropriate client libraries, programming language bindings (Python, Java, C, Ruby, …), CLI tools and their documentation are available in the application repository and the Knowledgebase.
The object interface is suitable for Third Party applications, high performance parallel access and scalability.
Users can access individual products, their parts, such as granules or metadata. They can even access parts of a single data file, which may be useful for applications such as individual tiles extraction from large jpeg2000 image files.
The Object API can be used by Third Party Services, but it is also the generic data access method used by other DIAS data access services and the EO Browser. It may also be used by registered Users to download data directly to their workstation.
While HTTP-based object interfaces are largely used in modern cloud applications, interactive human users and many legacy applications (including Sentinel Toolboxes) access data using a traditional POSIX filesystem interface. Usually, such users must first download the necessary files, store and access them from a local filesystem. This process is slow and requires additional temporary storage.
To address this challenge, CREODIAS makes the EO Data repository contents available directly through a POSIX filesystem interface. Users can access this interface from their Virtual Machines hosted on the CREODIAS infrastructure. Filesystem access may be achieved either through an emulation software that allows a Linux VM to ‘mount’ an object filestore as a filesystem or NFS proxy. In both cases, the mounted repository will be seen as a read-only file tree mounted under the user machine’s filesystem. Such mounting will be preconfigured on all VM templates, we will also provide detailed configuration instructions for those customers, who reinstall their systems on their own.
However filesystem interface is not recommended for processing chain applications and direct object access should be used whenever possible. The EO Data repository is an object storage and therefore any filesystem emulation or proxy lowers the performance and throughput of data access.
Please also note that the NFS interface has some limitations including very slow listing of directories which have thousands of objects in their tree (for example ls on daily production of Sentinel-2 may take substantial amount of time or even not complete due to protocol defined timeouts). Therefore it is recommended to go directly to product path which can be discovered using the product search tool:
EO Data Processing/Access HUB
The EO Data Access Hub provides a complete set of mechanisms to access EO Data as dynamically generated OGC standard web services. The Hub is focused on Copernicus satellites but also supporting other sources such as Landsat, PlanetLabs and others. Image data from the EO Data Repository is processed in real time using the CREODIAS cloud infrastructure and presented to the users in the form of customizable WMS/WMTS/WCS/WFS web services. It can be integrated into any mapping or web application, providing an easy-to-use and cost-effective way to exploit the data. It removes the major hassle of downloading, archiving and processing petabytes of data and simply makes the full and global repository easily available immediately via web services. Application developers can focus on added value services and end-user applications rather than having to deal with the complexity of remote sensing data.
The Hub itself is completed by front-ends to search, discover browse and download EO Data and to configure custom Web Service layers (EO Browser, Mosaic Generator, WMS Configurator). The EO Data Access Hub and its frontends are described in more detail in Section Data Related Services below.
Basic EO Data Access Hub services are free of charge - These are:
- WFS access to catalogue data
- WMS with true-color composite, full resolution; meant for use on end-user applications (e.g. QGIS), not for integration in web applications.
- EO Browser
- Mosaic Generator
More advanced functions are available with monthly or longer subscription:
- WMS with various configuration options
- WCS with various configuration options
- WMTS with various configuration options
The detailed price list is available in the Price List section.
Most of data indexed in the CREODIAS catalog (Locations: Local, Cached, Orderable) can be downloaded by Users with the CREODIAS Platform acting as a proxy. This functionality enables registered Users to download data indexed in the CREODIAS catalogue without the need to register or log-in to a remote data store. The CREODIAS acts as a proxy, downloading the desired product from its source and forwarding it to the User’s storage. The functionality can be used by CREODIAS Third Parties to download data into their environment on the CREODIAS Platform. It can also be used by CREODIAS Users to download data to their workstation over the Internet.
Proxy to External Copernicus Services
Some Copernicus Services (Location: External) are accessible as standalone web services that cannot be mirrored or otherwise copied to the CREODIAS Platform. Such Services will be represented in the CREODIAS Collections Catalog with a link to the original service. Depending on the access policy and interfacing capabilities of the original service provider, a http proxy or a single sign-on protocol will be used so that the User doesn’t have to login to the such remote system.