OpenStackClient is very useful tool to gain a powerful management of our projects in Command Line Interface. It implements new features and advantages: You can run the commands from the CLI, or include prepared scripts in Python to automate the functionality of your cloud storage. Moreover we could reach our OpenStack access on any computer if we inset credentials (username and password).

Eventually everyone may admit that generally OpenStackClient provides more opportunities to look into our compute facility deeply and much more precisely.

FAQ covers installation of Openstack Client under Ubuntu 18.04 LTS and Python 3. This method works on Ubuntu 20.04 as well.
Installation under other distributions may be simillar.

Let's start with the apt update && apt upgrade commands. This will update our system packages to the latest version

sudo apt update && sudo apt upgrade

Now let's install python3 and create a new python enviroment called simply "openstack_cli":

sudo apt install python3-venv
python3 -m venv openstack_cli
source openstack_cli/bin/activate

Now upgrade Python Package Installer (pip) to the latest version using command below:

pip install --upgrade pip

Finally install python-openstackclient:

pip install python-openstackclient

After this, you should be able to run openstack command from consone, eg:

openstack --help

If everything seems to work, we might firmly move on to our Horizon Panel.

Log in to your account.

Head straight to our email button in the upper-right-corner. Click on it.

Choose Openstack RC File v2 or v3.

Save your RC file on your disk.

If you use some text editor like vim it should consist of many variables related to your domain account like it:

Change your directory to downloading destination and execute the configuration file:

For example t will look like: cloud_xxxxx\project_without_eo-openrc.sh

Time to run this sh file:

. cloud_xxxxx\project_without_eo-openrc.sh

The prompted message will ask you for your domain password. Type it in and press enter.

If the password is correct then you've got access granted.

After that you can eg. check list of your servers by typing in console:

openstack server list

Output of this command should contain table with ID, Name, Status, Networks, Image and Flavor of your virtual machines.

We recommend checking Openstack documentation which contains lists of available commands. There you may reconsider many scenarios linked with your management.

v.2020-05-19

First of all, to use JupyterHub you have to be registered and signed in Creodias.

Please see: How to register to Creodias Portal?

After that, choose Tools →  JupyterHub from navigation menu:

Then you can see main page of JupyterHub. It is highly recommended to create new projects in persistent directory to avoid data loss caused by  e.g. by restart of your Jupyter Notebook container.

You can create a new script by choosing New and the language of your choice e.g. Python 3.

Product Query and selection

You can define search parameters and query the CREODIAS Finder engine for products that correspond to chosen parameters. Finder API is describerd here: 

https://creodias.eu/eo-data-finder-api-manual

To run the script you can use "Run" button on the top or use shortcut "Ctrl + Enter".

All shortcuts are available after clicking keyboard icon, there you can also define your own shortcuts. Keep in mind that you have to save your script.

Below you can find the above code.

Query = 'Tokio%20Winter' # here you can define semantic search
Cloud = '[0,5]' # here you can define cloud cover in percentage
sort = 'sortParam=cloudCover&sortOrder=ascending'
Coll = 'Sentinel2' # here you can choose which satellite are you interested in
ResultNumber = 0
OutputFileName = 'TrueColor'


URL = 'http://finder.creodias.eu/resto/api/collections/' + Coll +'/search.json?_pretty=true&q=' + Query + '&' + 'cloudCover=' + Cloud + '&' + sort import json, requests, os
products = json.loads(requests.get(URL).text)['features']
for product in products:
    print(product['properties']['productIdentifier'])
file = products[ResultNumber]['properties']['productIdentifier']

Product selection

Using selected product identifier (stored in "file" variable), you can open access bands to be later saved as a GeoTif.

import rasterio
path = file + '/GRANULE' + '/' + os.listdir(file + '/GRANULE')[0] + '/IMG_DATA'
if (os.path.isdir(path + '/' + sorted(os.listdir(path))[0])):
    path = path + '/' + sorted(os.listdir(path))[0]

band2 = rasterio.open(path + '/' + sorted(os.listdir(path))[1]) #blue
band3 = rasterio.open(path + '/' + sorted(os.listdir(path))[2]) #green
band4 = rasterio.open(path + '/' + sorted(os.listdir(path))[3]) #red
path

GeoTif creation

Now you can create a GeoTif file. Bands are stored in GeoTiff format and saved to the current directory of your Notebook.

We recommend storing the results in "persistent" directory.

from rasterio import plot
import matplotlib.pyplot as plt
trueColor = rasterio.open( OutputFileName + '.tiff','w',driver='Gtiff',
                         width=band4.width, height=band4.height,
                         count=3,
                         crs=band4.crs,
                         transform=band4.transform,
                         dtype=band4.dtypes[0]
                         )
trueColor.write(band2.read(1), 3) #blue
trueColor.write(band3.read(1), 2) #green
trueColor.write(band4.read(1), 1) #red
trueColor.close()
os.path.isfile(OutputFileName + '.tiff')

from rasterio.plot import show

fp = r'TrueColor.tiff'

img = rasterio.open(fp)
show(img)

In CREODIAS Finder you can search Sentinel-2 products by date and tile, rather than by geometry.

What exactly is a tile?

For Level-1C and Level-2A tiles are 100x100 km2 ortho-images in UTM/WGS84 projection.

Tile names correspond to UTM zone grid. The world is divided into 60 UTM zones of 8 degrees width in longitude, with numbers increasing towards the East. Tile names are included into products Identifiers, that allows fast searching for them.

To find an area of your interest, you can use Sentinel-2 tile grid KML file from ESA, that provides the footprint of all the tiles, and their names. This file is available under this link: Sentinel-2 tiles .kml file.

KML file can be easily opened in Google Earth Engine: https://earth.google.com/web/ as a new project.

Step by step guide is described here:  Import KML map data into Google Earth.

After that select the tile you're interested in and copy it's name.

For purposes of this guide we will use tile 33UYR, illustrated above (screenshot of Google Earth Engine) and below (screenshot of CREODIAS Finder).

Searching for products using FINDER.

The easiest way to reach products from tile is to copy the tile name into search criteria and confirm by 'Enter' key.

Here are all the results for the example tile:

To narrow down the total number of products, we can use a time frame filter, for example by inserting one month period to 'observed' date.

Searching for products using Rest query.

You can use "Rest query" field to modify your search:

To do that, copy/modify/paste the string from the field:

https://finder.creodias.eu/resto/api/collections/Sentinel2/search.json?maxRecords=10&productIdentifier=%2533UYR%25&sortParam=startDate&sortOrder=descending&status=all&dataset=ESA-DATASET

Tile name in our query is specified by the following keyword:

productIdentifier=%2533UYR%25

You can also specify time period for search,  by adding keywords 'startDate' and 'completionDate'.

For example add to your query, 'startDate=2022-01-17T00:00:00Z' and 'completionDate=2022-01-17T23:59:59Z', if you are searching for one day only.

https://finder.creodias.eu/resto/api/collections/Sentinel2/search.json?maxRecords=10&sortParam=startDate&sortOrder=descending&status=all&dataset=ESA-DATASET&productIdentifier=%2533UYR%25&startDate=2022-01-17T00:00:00Z&completionDate=2022-01-17T23:59:59Z

And the results of query:

Resources:

Download Sentinel-2 tile grid KML file:

https://sentinel.esa.int/documents/247904/1955685/S2A_OPER_GIP_TILPAR_MPC__20151209T095117_V20150622T000000_21000101T000000_B00.kml

Articles you may find helpful:

https://labo.obs-mip.fr/multitemp/the-sentinel-2-tiles-how-they-work/

https://sentinels.copernicus.eu/web/sentinel/user-guides/sentinel-2-msi/naming-convention

https://support.google.com/earth/answer/7365595?hl=en&co=GENIE.Platform%3DDesktop

How to delete Order / Order Product

Using Finder API you can delete an entire order or just individual products.

To do that please make sure that you have Openstack CLI installed on your operating system. You have to also obtain KeyCloak.

How to install Openstack Client on Linux

How to install Openstack Client on Windows

How to obtain KeyCloak Token / Order products using API

Deleting an entire order

To delete an entire order you can use following command: "curl -X DELETE https://finder.creodias.eu/api/order/*order_ID*/ -H "Content-Type: application/json" -H "Keycloak-Token: ${KEYCLOAK_TOKEN}" | python3 -m json.tool"

Example for Order 1319047

llesiak@llesiakNB:~$ curl -X DELETE https://finder.creodias.eu/api/order/1319047/ -H "Content-Type: application/json" -H "Keycloak-Token: ${KEYCLOAK_TOKEN}" | python3 -m json.tool

As you can see the whole order has been deleted (canceled).

Deleting Order Products

To delete an product you'll need product item ID.

To get ID of Item that you've ordered please use this request: "curl -X GET https://finder.creodias.eu/api/order/*order_ID*/order_items -H "Content-Type: application/json" -H "Keycloak-Token: ${KEYCLOAK_TOKEN}" | python3 -m json.tool"

As a result you'll get details of an order. Example for order 1317258.

llesiak@llesiakNB:~$ curl -X GET https://finder.creodias.eu/api/order/1317258/order_items -H "Content-Type: application/json" -H "Keycloak-Token: ${KEYCLOAK_TOKEN}" | python3 -m json.tool
{
    "items": [
        {
            "id": 2491216,
            "name": "S2A_MSIL1C_20190228T110001_N0207_R094_T31VEG_20190228T111324",
            "status": "done",
            "add_date": "2022-01-14T14:05:59.329727+00:00",
            "mod_date": "2022-01-14T14:06:05.849253+00:00",
            "extra_info": "",
            "result": {
                "processed_name": "S2A_MSIL2A_20190228T110001_N0211_R094_T31VEG_20190228T113827",
                "processed_size": 673655801,
                "processing_metadata": {}
            },
            "order": {
                "extra": null
            }
        }
    ],
    "total": 1,
    "page": 0,
    "size": 10
}

After getting product ID you can delete it using this command: "curl -X DELETE https://finder.creodias.eu/api/order_item/*order_Item_ID*-H "Content-Type: application/json" -H "Keycloak-Token: ${KEYCLOAK_TOKEN}" | python3 -m json.tool"

Example for product 2493999.

llesiak@llesiakNB:~$ curl -X DELETE https://finder.creodias.eu/api/order_item/2493999 -H "Content-Type: application/json" -H "Keycloak-Token: ${KEYCLOAK_TOKEN}" | python3 -m json.tool
{
    "id": 2493999,
    "name": "S2A_MSIL2A_20220113T194651_N0301_R099_T07LCL_20220113T214508.SAFE",
    "status": "cancelled",
    "add_date": "2022-01-18T14:25:00.387474+00:00",
    "mod_date": "2022-01-18T17:27:17.144279+00:00",
    "extra_info": "",
    "result": {
        "processed_name": "S2A_MSIL2A_20220113T194651_N0301_R099_T07LCL_20220113T214508",
        "processed_size": 153890883,
        "processing_metadata": null
    },
    "order": {
        "extra": null
    }
}

As you can see only one product has been deleted.

In order to start ordering products using Finder API you may use cURL library that is mainly responsible for sending HTTP requests. By default it is preinstalled on each virtual machine.

Information

Client is able to perform one POST operation:

• order processed products: Sentinel-2 products are available in processing level 1. By ordering processed product we send a request to obtain a  product in processing level 2.
• order external products: Product is visible in EO Finder but is not available in our repositories yet. Its status is recognized as "Orderable".

Documentation

For comprehensive API description you can visit this page: Finder API

Ordering processed product

Syntax:

curl -X POST https://finder.creodias.eu/api/order/ -H 'Content-Type:application/json' -H "Keycloak-Token: ${KEYCLOAK_TOKEN}" \
  -d '{"priority":number,
  "order_name":"name",
  "processor": "sen2cor",
  "identifier_list":["title_without_safe_extension"]}' | python -m json.tool

Title is a representative fullname of the product in our repository. In the identifier_list parameter above you have to place title without its extension SAFE extension e.g S2A_MSIL1C_20190521T063631_N0207_R120_T40RES_20190521T094626.

Example for one product from Sentinel-2 collection:

curl -X POST https://finder.creodias.eu/api/order/ -H 'Content-Type:application/json' -H "Keycloak-Token: ${KEYCLOAK_TOKEN}" \
  -d '{"priority":1,
  "order_name":"test_order_for_FAQ",
  "processor": "sen2cor",
  "identifier_list":["S2A_MSIL1C_20190228T110001_N0207_R094_T31VEG_20190228T111324"]}' | python -m json.tool

Checking the order list

Syntax:

curl -X GET https://finder.creodias.eu/api/order/ -H "Content-Type: application/json" -H "Keycloak-Token: ${KEYCLOAK_TOKEN}" | python -m json.tool

Verifying the status of a particular order

Syntax:

curl -X GET https://finder.creodias.eu/api/order/order_id/order_items/ -H "Content-Type: application/json" -H "Keycloak-Token: ${KEYCLOAK_TOKEN}" | python -m json.tool

Example for order with ID 203:

curl -X GET https://finder.creodias.eu/api/order/203/order_items/ -H "Content-Type: application/json" -H "Keycloak-Token: ${KEYCLOAK_TOKEN}" | python -m json.tool

e.g

curl -X GET https://finder.creodias.eu/api/order/86283/order_items/?page=2 \
-H "Content-Type: application/json" -H "Keycloak-Token: ${KEYCLOAK_TOKEN}" | python -m json.tool

You can see a total count number on every GET query in the first row:

"total": 89,
"page": 0,
"size": 10                                       

The API will no longer provide "previous" and "next" URLs. The user should iterate over the results until the number of items in the response is smaller than "size" or the following condition is met: "size" x "page" >= "total".

Obtain a KeyCloak token

export KEYCLOAK_TOKEN=$(curl -d 'client_id=CLOUDFERRO_PUBLIC' \
                             -d "username=${OS_USERNAME}" \
                             -d "password=${OS_PASSWORD}" \
                             -d 'grant_type=password' \
                             'https://auth.creodias.eu/auth/realms/DIAS/protocol/openid-connect/token' | \
                             python -m json.tool | grep "access_token" | awk -F\" '{print $4}')

Here you can locate the button to proceed download in Horizon panel:

You can also obtain token without using Python environment and authenticating with OpenStack RC File using simpler curl command:

curl \
   -d 'client_id=CLOUDFERRO_PUBLIC' \
   -d 'username=<your_CREODIAS_username>' \
   -d 'password=<your_CREODIAS_password>' \
   -d 'grant_type=password' \
   'https://auth.creodias.eu/auth/realms/DIAS/protocol/openid-connect/token' \
   | python -m json.tool | grep "access_token"

Please note that using this curl command will not save your token in environmental variables and requires entering your username and password.

Troubleshooting

In the case of failure you may encounter those symptoms:

A)

{"ErrorMessage":"orderProduct- Forbidden","ErrorCode":403}

printenv KEYCLOAK_TOKEN

{"ErrorMessage":"processModuleRoute - Not Found","ErrorCode":404}

{
    "error": "invalid_grant",
    "error_description": "Invalid user credentials"
}