Your Processing Environment

EOData Catalogue API Manual

Catalogue API is a new data catalogue API with Odata and OpenSearch interface. 

OData

Query structure

As a general note, Odata query consistst of elements which in this documentation are called "options". Interface supports the following search options:

• filter
• orderby
• top
• skip
• count
• expand

Search options should always be preceded with $ and consecutive options should be separated with &.

Consecutive filters within filter option should be separated with and or or. Not operator can also be used e.g.:

https://datahub.creodias.eu/odata/v1/Products?$filter=not contains(Name,'S2') and ContentDate/Start gt 2022-05-03T00:00:00.000Z and ContentDate/Start lt 2022-05-03T00:10:00.000Z&$orderby=ContentDate/Start&$top=100

Performance note:

To accelerate the query performance it is recommended to limit the query by acquisition dates e.g.:

ContentDate/Start gt 2022-05-03T00:00:00.000Z and ContentDate/Start lt 2022-05-21T00:00:00.000Z

Filter option

Query by name

To search for a specific product by its exact name:

https://datahub.creodias.eu/odata/v1/Products?$filter=Name eq 'S1A_IW_GRDH_1SDV_20141031T161924_20141031T161949_003076_003856_634E.SAFE'

To search for products containing “S1A” in their names:

https://datahub.creodias.eu/odata/v1/Products?$filter=contains(Name,'S1A') and ContentDate/Start gt 2022-05-03T00:00:00.000Z and ContentDate/Start lt 2022-05-21T00:00:00.000Z

Alternatively to contains, endswith and startswith can be used, to search for products ending or starting with provided string.

Query by list

In case a user desires to search for multiple products by name in one query, POST method can be used:

POST

https://datahub.creodias.eu/odata/v1/Products/OData.CSC.FilterList

Request body:

{
    "FilterProducts":
    [
        {"Name": "S1A_IW_GRDH_1SDV_20141031T161924_20141031T161949_003076_003856_634E.SAFE"},
        {"Name": "S3B_SL_1_RBT____20190116T050535_20190116T050835_20190117T125958_0179_021_048_0000_LN2_O_NT_003.SEN3"},
        {"Name": "xxxxxxxx.06.tar"}
    ]
}

Two results are returned, as there is no product named xxxxxxxx.06.tar.

Query Collection of Products

To search for products within a specific collection:

https://datahub.creodias.eu/odata/v1/Products?$filter=Collection/Name eq 'SENTINEL-2' and ContentDate/Start gt 2022-05-03T00:00:00.000Z and ContentDate/Start lt 2022-05-03T00:11:00.000Z

The following collections are currently available:

• SENTINEL-1
• SENTINEL-2
• SENTINEL-3
• SENTINEL-5P
• SENTINEL-6
• SENTINEL-1-RTC
• LANDSAT-5
• LANDSAT-7
• LANDSAT-8
• SMOS
• TERRAAQUA
• COP-DEM
• ENVISAT
• S2GLC

Query by Publication Date

To search for products published between two dates:

https://datahub.creodias.eu/odata/v1/Products?$filter=PublicationDate gt 2019-05-15T00:00:00.000Z and PublicationDate lt 2019-05-16T00:00:00.000Z

To define inclusive interval ge and le parameters can be used:

https://datahub.creodias.eu/odata/v1/Products?$filter=PublicationDate ge 2019-05-15T00:00:00.000Z and PublicationDate le 2019-05-16T00:00:00.000Z

Query by Sensing Date

To search for products acquired between two dates:

https://datahub.creodias.eu/odata/v1/Products?$filter=ContentDate/Start gt 2019-05-15T00:00:00.000Z and ContentDate/Start lt 2019-05-16T00:00:00.000Z

Usually, there are two parameters describing the ContentDate (Acquisition Dates) for a product - Start and End. Depending on what the user is looking for, these parameters can be mixed, e.g.:

https://datahub.creodias.eu/odata/v1/Products?$filter=ContentDate/Start gt 2019-05-15T00:00:00.000Z and ContentDate/End lt 2019-05-15T00:05:00.000Z

Query by Geographic Criteria

To search for products intersecting the specified polygon:

https://datahub.creodias.eu/odata/v1/Products?$filter=OData.CSC.Intersects(area=geography'SRID=4326;POLYGON((12.655118166047592 47.44667197521409,21.39065656328509 48.347694733853245,28.334291357162826 41.877123516783655,17.47086198383573 40.35854475076158,12.655118166047592 47.44667197521409))') and ContentDate/Start gt 2022-05-20T00:00:00.000Z and ContentDate/Start lt 2022-05-21T00:00:00.000Z

To search for products intersecting the specified point:

https://datahub.creodias.eu/odata/v1/Products?$filter=OData.CSC.Intersects(area=geography%27SRID=4326;POINT(-0.5319577002158441%2028.65487836189358)%27)

Disclaimers:

1. MULTIPOLYGON is currently not supported.
2. Polygon must start and end with the same point.
3. Coordinates must be given in EPSG 4326

Query by attributes

To search for products by attributes it is necessary to build a filter with the following structure:

Attributes/OData.CSC.ValueTypeAttribute/any(att:att/Name eq '[Attribute.Name]' and att/OData.CSC.ValueTypeAttribute/Value eq '[Attribute.Value]')

where

• ValueTypeAttribute can take the following values:
  • StringAttribute
  • DoubleAttribute
  • IntegerAttribute
  • DateTimeOffsetAttribute
• [Attribute.Name] is the attribute name which can take multiple values, depending on collection (Attachment 1 - Coming soon)
• eq before [Attribute.Value] can be substituted with le, lt, ge, gt in case of Integer, Double or DateTimeOffset Attributes
• [Attribute.Value] is the specific value that the user is searching for

To get products with cloudCover<40% between two dates:

https://datahub.creodias.eu/odata/v1/Products?$filter=Attributes/OData.CSC.DoubleAttribute/any(att:att/Name eq 'cloudCover' and att/OData.CSC.DoubleAttribute/Value le 40.00) and ContentDate/Start gt 2022-01-01T00:00:00.000Z and ContentDate/Start lt 2022-01-03T00:00:00.000Z&$top=10

To get products with cloudCover< 10% and productType=S2MSI2A and ASCENDING orbitDirection between two dates:

https://datahub.creodias.eu/odata/v1/Products?$filter=Attributes/OData.CSC.DoubleAttribute/any(att:att/Name eq 'cloudCover' and att/OData.CSC.DoubleAttribute/Value lt 10.00) and Attributes/OData.CSC.StringAttribute/any(att:att/Name eq 'productType' and att/OData.CSC.StringAttribute/Value eq 'S2MSI2A') and Attributes/OData.CSC.StringAttribute/any(att:att/Name eq 'orbitDirection' and att/OData.CSC.StringAttribute/Value eq 'ASCENDING') and ContentDate/Start gt 2022-05-03T00:00:00.000Z and ContentDate/Start lt 2022-05-03T04:00:00.000Z&$top=10

Orderby option

Orderby option can be used to order the products in an ascending (asc) or descending (desc) direction. If asc or desc not specified, then the resources will be ordered in ascending order.

To order products by ContentDate/Start in a descending direction:

https://datahub.creodias.eu/odata/v1/Products?$filter=contains(Name,'S1A_EW_GRD') and ContentDate/Start gt 2022-05-03T00:00:00.000Z and ContentDate/Start lt 2022-05-03T03:00:00.000Z&$orderby=ContentDate/Start des

By default, if orderby option is not used, the results are not ordered. If orderby option is used, additional orderby by id is also used, so that the results are fully ordered and no products are lost while paginating through the results.

The acceptable arguments for this option: ContentDate/Start, ContentDate/End, PublicationDate, ModificationDate, in directions: asc, desc

Top option

Top option specifies the maximum number of items returned from a query. 

To limit the number of results:

https://datahub.creodias.eu/odata/v1/Products?$filter=contains(Name,%27S1A_EW_GRD%27)%20and%20ContentDate/Start%20gt%202022-05-03T00:00:00.000Z%20and%20ContentDate/Start%20lt%202022-05-03T12:00:00.000Z&$top=100

The default value is set to 20.

The acceptable arguments for this option: Integer <0,1000>

Skip option

Skip option can be used to skip a specific number of results. Exemplary application of this option would be paginating through the results, however for performance reasons, we recommend limiting queries with small time intervals as a substitute of using skip in a more generic query.

To skip a specific number of results:

https://datahub.creodias.eu/odata/v1/Products?$filter=contains(Name,%27S1A_EW_GRD%27)%20and%20ContentDate/Start%20gt%202022-05-03T00:00:00.000Z%20and%20ContentDate/Start%20lt%202022-05-03T12:00:00.000Z&$skip=23

The default value is set to 0. 

Whenever a query results in more products than 20 (default top value), the API provides a nextLink at the bottom of the page:

"@odata.nextLink": "http://datahub.creodias.eu/odata/v1/Products?$filter=contains(Name,'S1A_EW_GRD')+and+ContentDate/Start+gt+2022-05-03T00:00:00.000Z+and+ContentDate/Start+lt+2022-05-03T12:00:00.000Z&$skip=20"

The acceptable arguments for this option: Integer <0,10000>

Count option

Count option enables users to get the exact number of products matching the query. This option is disabled by default to accelerate the query performance.

To get the exact number of products for a given query:

https://datahub.creodias.eu/odata/v1/Products?$filter=contains(Name,%27S1A_EW_GRD%27)%20and%20ContentDate/Start%20gt%202022-05-03T00:00:00.000Z%20and%20ContentDate/Start%20lt%202022-05-03T12:00:00.000Z&$count=True

The acceptable arguments for this option: True, true, 1, False, false, 0

Expand option

Expand option enables users to see full metadata of each returned result.

To see the metadata of the results:

https://datahub.creodias.eu/odata/v1/Products?$filter=contains(Name,%27S1A_EW_GRD%27)%20and%20ContentDate/Start%20gt%202022-05-03T00:00:00.000Z%20and%20ContentDate/Start%20lt%202022-05-03T12:00:00.000Z&$expand=Attributes

The acceptable arguments for this option: Attributes

Product Download

To download products:

Where Id is an Id of the product returned by the search query, e.g.:

https://datahub.creodias.eu/odata/v1/Products(a6212de3-f2e4-58c2-840b-7f42c3c8c612)/$value

Only authorized users are allowed to download products

To get the token:

KEYCLOAK_TOKEN=$(curl -s --location --request POST 'https://identity.cloudferro.com/auth/realms/<BRAND>/protocol/openid-connect/token' \
--data-urlencode 'grant_type=password' \
--data-urlencode 'username=<USER>' \
--data-urlencode 'password=<PASSWORD>' \
--data-urlencode 'client_id=CLOUDFERRO_PUBLIC'|jq .access_token|tr -d '"')

or

' -d 'password=' -d 'grant_type=password' 'https://identity.cloudferro.com/auth/realms//protocol/openid-connect/token' | python -m json.tool | grep "access_token" | awk -F\" '{print $4}')]]>

Where USER and PASSWORD are credentials to Your CloudFerro account in specific BRAND. Brand names are listed below with API from which You can get your token.

To download the product:

curl -H "Authorization: Bearer $KEYLOAK_TOKEN" 'https://datahub.creodias.eu/odata/v1/Products(060882f4-0a34-5f14-8e25-6876e4470b0d)/$value' --output /tmp/product.zip

or

wget  --header "Authorization: Bearer $KEYCLOAK_TOKEN" 'http://datahub.creodias.eu/odata/v1/Products(db0c8ef3-8ec0-5185-a537-812dad3c58f8)/$value' -O example_odata.zip

To download a single file by ODATA API:

wget --header "Authorization: Bearer ${KEYCLOAK_TOKEN}" 'http://datahub.creodias.eu/odata/v1/Products(db0c8ef3-8ec0-5185-a537-812dad3c58f8)/Nodes(S2A_MSIL1C_20180927T051221_N0206_R033_T42FXL_20180927T073143.SAFE)/Nodes(manifest.safe)/$value' -O manifest.safe --no-check-certificate 

Obtaining token for Creodias New

Two-factor authentication is enabled on Creodias New. It means that you need to add an additional parameter for obtaining the Keycloak token.

You should use the following code for that purpose instead of the one provided above:

KEYCLOAK_TOKEN=$(curl -d 'client_id=CLOUDFERRO_PUBLIC' -d 'username=<USER>' -d 'password=<PASSWORD>' -d 'grant_type=password' -d 'totp=<TOTP>' 'https://identity.cloudferro.com/auth/realms/Creodias-new/protocol/openid-connect/token' | python3 -m json.tool | grep "access_token" | awk -F\" '{print $4}')

As previously, replace <USER> and <PASSWORD> with credentials to your CloudFerro account. This time, you need to also replace <TOTP> with the TOTP six-digit code generated by your two-factor authentication application.

The instructions for downloading EODATA do not change.

OpenSearch

Using OpenSearch interface to query Data Catalogue

All queries may be executed as simple HTTP-Get calls, by typing the query in web browser address line, by using any HTTP client, e.g. curl or wget, or from inside of users’ program. The database is accessible free and anonymously (open for anonymous access for everyone, no authorization is used) It may be accessed both from the internal network (virtual machines in Creodias) and from outside, e.g. your home computer. Note, that the actual EO data themselves are restricted to authorized users, only the Data Catalogue is open.

General Rules

The queries produce results in JSON format. Base url:

http://datahub.creodias.eu/resto/api/collections/search.json?

Most queries are case-sensitive.

Collections

The data are organized in so-called collections, corresponding to various satellites. A query may search for data in all collections, or in one particular collection only. If only one satellite is in the field of interest, the second approach is faster and more efficient, than filtering the general query. For example, to find 10 most recent Sentinel-2 products with cloud cover below 10%, the query should look like:

$ wget -O - "http://datahub.creodias.eu/resto/api/collections/Sentinel2/search.json?cloudCover=[0,10]&startDate=2022-06-11&completionDate=2022-06-22&maxRecords=10"

while if the collection field is missing in the URL, the products from all the satellites are returned:

$ wget -O - "https://datahub.creodias.eu/resto/api/collections/search.json?cloudCover=[0,10]&startDate=2022-06-11&completionDate=2022-06-22&maxRecords=10"

As for today the following collections are defined and may be used:

Sentinel1 or SENTINEL-1

Sentinel2 or SENTINEL-2

Sentinel3 or SENTINEL-3

Sentinel5P or SENTINEL-5P

Sentinel6 or SENTINEL-6

Landsat8 or LANDSAT-8

Landsat7 or LANDSAT-7

Landsat5 or LANDSAT-5

Envisat or ENVISAT

Note, that collection names vary a bit from satellite names, as they are used in EO Data repository. For example, the collection is named Sentinel2, while in the repository its data are located within /eodata/Sentinel-2/.... branch of the repository tree.

Output sorting and limiting

By default, maximum 20 products are returned only. You may change the limit (beware of long execution time for queries about thousands of products) using the phrase

maxRecords=nnn

If the query is very general and the number of matching products is large, the next pages of products may be retrieved

page=nnn

You may also change the order of how the products are presented, using the phrase like

sortParam=startDate

will sort the output by observation date. The following orderings are implemented:

startDate - the date when the observation was made (start)

completionDate - the date when the observation was made (end)

published - the date when the product got published in our repository

each of them may be accompanied by

sortOrder=ascending or sortOrder=descending

For example the query

http://datahub.creodias.eu/resto/api/collections/Sentinel2/search.json?startDate=2021-07-01&completionDate=2021-07-31&sortParam=startDate&maxRecords=20

will return 20 products from July 2021, while the next query would return the next 20:

http://datahub.creodias.eu/resto/api/collections/Sentinel2/search.json?startDate=2021-07-01&completionDate=2021-07-31&sortParam=startDate&maxRecords=20&page=2

Formal queries

The formal query is invoked as a sequence of sub phrases, separated by &. The result is a conjunction of all sub phrases. It is impossible to use an alternative in the question. The query must be specified as a formal query. 

The example of formal query - about cloudless (cloud cover lower or equal to 10%) products for a specific location:

https://datahub.creodias.eu/resto/api/collections/Sentinel2/search.json?cloudCover=[0,10]&startDate=2021-06-21&completionDate=2021-09-22&lon=21.01&lat=52.22

The queries are in form param=value or param=[minvalue,maxvalue]. Most of the parameters are common for all collections, but some are specific for some them (e.g. cloudCover applies to optical satellites, but polarisation applies to radar ones), or just single one.

Geography and time-frame

The common set of parameters are:

startDate, completionDate - the date limits of the observation. The time may also be specified, e.g. 2021-10-01T21:30:00Z 

publishedAfter, publishedBefore - the date limits when the product was published in our repository

lon, lat - geographical position, expressed in military style (EPSG:4326, as decimal fraction of degrees, positive for eastern latitude and northern longitude)
radius - region of interest, defined as a circle with centre in point determined by the longitude and latitude with radius expressed in meters (it won't work with point manually selected in EOFinder/Data Explorer)

geometry - region of interest, defined as WKT string (POINT, POLYGON, etc.)

box - region of interest, defined as the rectangle with given (west,south,east,north) values

Volatile features

Some terrain-like feature masks are not permanent but describing a single scene only. The most commonly used such feature is cloudiness, or cloudCover, which is defined for most of the products coming from optical sensors. For example:

cloudCover=[0,10]

selects only those scenes, which are covered by clouds by no more than 10%.

Caution: to be meaningful, the cloudiness must be provided with each product, while in many products is missing. If the cloudiness is unknown for the scene, it is marked by a value of 0 or -1. cloudCover=0 is therefore ambiguous: it may either mean totally cloudless sky or the cloudy scene for which cloud cover had not been estimated during original data processing.

Satellite features

instrument - meaningful only for satellites equipped with multiple instruments. The possible values are satellite specific.

productType - the actual types possible are specific for every satellite. 

sensorMode - also satellite and sensor specific. E.g. (for Sentinel-1): sensorMode=EW

orbitDirection - ASCENDING or DESCENDING. For most heliosynchronous satellites descending orbits means the day scenes, while ascending means night ones. For many optical satellites (e.g. Sentinel-2) only day scenes are published.

resolution - expected spatial resolution of the product defined in meters.

status:

• ONLINE

• OFFLINE 

Some additional parameters are strictly satellite-specific, e.g. polarisation, which is defined only for Sentinel-1

For every satellite (collection) its set of query-able parameters may be obtained by a query like:

https://datahub.creodias.eu/resto/api/collections/Sentinel1/describe.xml

The resulting XML file provides full list of the parameters for the collection, with their very brief descriptions.

Order API Documentation

For comprehensive Order API description you can visit these pages::

EO Data Ordering API2 Manual