TAGS

There are no tags associated with this article.

How to install OpenStackClient (Linux)?

OpenStackClient is very useful tool to gain a powerful management of our projects in Command Line Interface. It implements new features and advantages: You can run the commands from the CLI, or include prepared scripts in Python to automate the functionality of your cloud storage. Moreover we could reach our OpenStack access on any computer if we inset credentials (username and password).

Eventually everyone may admit that generally OpenStackClient provides more opportunities to look into our compute facility deeply and much more precisely.

 

Attention:

It is strongly recommended to use virtual environments which do not affect system variables globally. If something goes wrong, everything is happening in separated space.

 

FAQ covers installation of Openstack Client under Ubuntu 18.04 LTS and Python 3. This method works on Ubuntu 20.04 as well.
Installation under other distributions may be simillar.

 

Let's start with the apt update && apt upgrade commands. This will update our system packages to the latest version

sudo apt update && sudo apt upgrade

 

Now let's install python3 and create a new python enviroment called simply "openstack_cli":

sudo apt install python3-venv
python3 -m venv openstack_cli
source openstack_cli/bin/activate

 

Now upgrade Python Package Installer (pip) to the latest version using command below:

pip install --upgrade pip

 

Finally install python-openstackclient:

pip install python-openstackclient

 

After this, you should be able to run openstack command from consone, eg:

openstack --help

 

If everything seems to work, we might firmly move on to our Horizon Panel.

Log in to your account.

Head straight to our email button in the upper-right-corner. Click on it.

 

Choose Openstack RC File v2 or v3.

Save your RC file on your disk.

If you use some text editor like vim it should consist of many variables related to your domain account like it:

Change your directory to downloading destination and execute the configuration file:

For example t will look like: cloud_xxxxx\project_without_eo-openrc.sh

Time to run this sh file:

. cloud_xxxxx\project_without_eo-openrc.sh

 

The prompted message will ask you for your domain password. Type it in and press enter.

If the password is correct then you've got access granted.

 

After that you can eg. check list of your servers by typing in console:

openstack server list

Output of this command should contain table with ID, Name, Status, Networks, Image and Flavor of your virtual machines.

We recommend checking Openstack documentation which contains lists of available commands. There you may reconsider many scenarios linked with your management.

 

 

v.2020-05-19


How to use JupyterHub?

Due to improving safety and security, CREODIAS introduce maintenance works on the JupyterHub. The service will not have access to internet. In justified cases, we can provide internet access for 24 hours. If you need such access, please contact us at support@creodias.eu.

First of all, to use JupyterHub you have to be registered and signed in Creodias.

Please see: How to register to Creodias Portal?

After that, choose Tools →  JupyterHub from navigation menu:

 

Then you can see main page of JupyterHub. It is highly recommended to create new projects in persistent directory to avoid data loss caused by  e.g. by restart of your Jupyter Notebook container.

 

You can create a new script by choosing New and the language of your choice e.g. Python 3.

 

Product Query and selection

You can define search parameters and query the CREODIAS Finder engine for products that correspond to chosen parameters. Finder API is describerd here: 

https://creodias.eu/eo-data-finder-api-manual

 

To run the script you can use "Run" button on the top or use shortcut "Ctrl + Enter".

All shortcuts are available after clicking keyboard icon, there you can also define your own shortcuts. Keep in mind that you have to save your script.

 

Below you can find the above code.

Query = 'Tokio%20Winter' # here you can define semantic search
Cloud = '[0,5]' # here you can define cloud cover in percentage
sort = 'sortParam=cloudCover&sortOrder=ascending'
Coll = 'Sentinel2' # here you can choose which satellite are you interested in
ResultNumber = 0
OutputFileName = 'TrueColor'


URL = 'http://finder.creodias.eu/resto/api/collections/' + Coll +'/search.json?_pretty=true&q=' + Query + '&' + 'cloudCover=' + Cloud + '&' + sort import json, requests, os
products = json.loads(requests.get(URL).text)['features']
for product in products:
    print(product['properties']['productIdentifier'])
file = products[ResultNumber]['properties']['productIdentifier']

 

Product selection

Using selected product identifier (stored in "file" variable), you can open access bands to be later saved as a GeoTif.

import rasterio
path = file + '/GRANULE' + '/' + os.listdir(file + '/GRANULE')[0] + '/IMG_DATA'
if (os.path.isdir(path + '/' + sorted(os.listdir(path))[0])):
    path = path + '/' + sorted(os.listdir(path))[0]

band2 = rasterio.open(path + '/' + sorted(os.listdir(path))[1]) #blue
band3 = rasterio.open(path + '/' + sorted(os.listdir(path))[2]) #green
band4 = rasterio.open(path + '/' + sorted(os.listdir(path))[3]) #red
path

 

GeoTif creation

Now you can create a GeoTif file. Bands are stored in GeoTiff format and saved to the current directory of your Notebook.

We recommend storing the results in "persistent" directory.

from rasterio import plot
import matplotlib.pyplot as plt
trueColor = rasterio.open( OutputFileName + '.tiff','w',driver='Gtiff',
                         width=band4.width, height=band4.height,
                         count=3,
                         crs=band4.crs,
                         transform=band4.transform,
                         dtype=band4.dtypes[0]
                         )
trueColor.write(band2.read(1), 3) #blue
trueColor.write(band3.read(1), 2) #green
trueColor.write(band4.read(1), 1) #red
trueColor.close()
os.path.isfile(OutputFileName + '.tiff')

 

Displaying the image

Now you can display the image.

from rasterio.plot import show

fp = r'TrueColor.tiff'

img = rasterio.open(fp)
show(img)

 

Now you can download the result to your desktop.

 


How to search for satellite products using tiles?

 

In CREODIAS Finder you can search Sentinel-2 products by date and tile, rather than by geometry.

 

What exactly is a tile?

 

For Level-1C and Level-2A tiles are 100x100 km2 ortho-images in UTM/WGS84 projection.

Tile names correspond to UTM zone grid. The world is divided into 60 UTM zones of 8 degrees width in longitude, with numbers increasing towards the East. Tile names are included into products Identifiers, that allows fast searching for them.

To find an area of your interest, you can use Sentinel-2 tile grid KML file from ESA, that provides the footprint of all the tiles, and their names. This file is available under this link: Sentinel-2 tiles .kml file.

KML file can be easily opened in Google Earth Engine: https://earth.google.com/web/ as a new project.

Step by step guide is described here:  Import KML map data into Google Earth.

After that select the tile you're interested in and copy it's name.

 

For purposes of this guide we will use tile 33UYR, illustrated above (screenshot of Google Earth Engine) and below (screenshot of CREODIAS Finder).

 

Searching for products using FINDER.

The easiest way to reach products from tile is to copy the tile name into search criteria and confirm by 'Enter' key.

 

Here are all the results for the example tile:

 

To narrow down the total number of products, we can use a time frame filter, for example by inserting one month period to 'observed' date.

 

Searching for products using Rest query.

You can use "Rest query" field to modify your search:

 

To do that, copy/modify/paste the string from the field:

https://finder.creodias.eu/resto/api/collections/Sentinel2/search.json?maxRecords=10&productIdentifier=%2533UYR%25&sortParam=startDate&sortOrder=descending&status=all&dataset=ESA-DATASET

 

Tile name in our query is specified by the following keyword:

productIdentifier=%2533UYR%25

 

You can also specify time period for search,  by adding keywords 'startDate' and 'completionDate'.

For example add to your query, 'startDate=2022-01-17T00:00:00Z' and 'completionDate=2022-01-17T23:59:59Z', if you are searching for one day only.

https://finder.creodias.eu/resto/api/collections/Sentinel2/search.json?maxRecords=10&sortParam=startDate&sortOrder=descending&status=all&dataset=ESA-DATASET&productIdentifier=%2533UYR%25&startDate=2022-01-17T00:00:00Z&completionDate=2022-01-17T23:59:59Z

 

And the results of query:

 

 

 

Resources:

Download Sentinel-2 tile grid KML file:

https://sentinel.esa.int/documents/247904/1955685/S2A_OPER_GIP_TILPAR_MPC__20151209T095117_V20150622T000000_21000101T000000_B00.kml

Articles you may find helpful:

https://labo.obs-mip.fr/multitemp/the-sentinel-2-tiles-how-they-work/

https://sentinels.copernicus.eu/web/sentinel/user-guides/sentinel-2-msi/naming-convention

https://support.google.com/earth/answer/7365595?hl=en&co=GENIE.Platform%3DDesktop

 


How to delete Finder Order / Order Product via API

How to delete Order / Order Product

Using Finder API you can delete an entire order or just individual products.

To do that please make sure that you have Openstack CLI installed on your operating system. You have to also obtain KeyCloak.

How to install Openstack Client on Linux

How to install Openstack Client on Windows

How to obtain KeyCloak Token / Order products using API

 

Deleting an entire order

To delete an entire order you can use following command: "curl -X DELETE https://finder.creodias.eu/api/order/*order_ID*/ -H "Content-Type: application/json" -H "Keycloak-Token: ${KEYCLOAK_TOKEN}" | python3 -m json.tool"

Example for Order 1319047

llesiak@llesiakNB:~$ curl -X DELETE https://finder.creodias.eu/api/order/1319047/ -H "Content-Type: application/json" -H "Keycloak-Token: ${KEYCLOAK_TOKEN}" | python3 -m json.tool

As you can see the whole order has been deleted (canceled).

 

 

Deleting Order Products

To delete an product you'll need product item ID.

To get ID of Item that you've ordered please use this request: "curl -X GET https://finder.creodias.eu/api/order/*order_ID*/order_items -H "Content-Type: application/json" -H "Keycloak-Token: ${KEYCLOAK_TOKEN}" | python3 -m json.tool"

As a result you'll get details of an order. Example for order 1317258.

llesiak@llesiakNB:~$ curl -X GET https://finder.creodias.eu/api/order/1317258/order_items -H "Content-Type: application/json" -H "Keycloak-Token: ${KEYCLOAK_TOKEN}" | python3 -m json.tool
{
    "items": [
        {
            "id": 2491216,
            "name": "S2A_MSIL1C_20190228T110001_N0207_R094_T31VEG_20190228T111324",
            "status": "done",
            "add_date": "2022-01-14T14:05:59.329727+00:00",
            "mod_date": "2022-01-14T14:06:05.849253+00:00",
            "extra_info": "",
            "result": {
                "processed_name": "S2A_MSIL2A_20190228T110001_N0211_R094_T31VEG_20190228T113827",
                "processed_size": 673655801,
                "processing_metadata": {}
            },
            "order": {
                "extra": null
            }
        }
    ],
    "total": 1,
    "page": 0,
    "size": 10
}

 

After getting product ID you can delete it using this command: "curl -X DELETE https://finder.creodias.eu/api/order_item/*order_Item_ID*-H "Content-Type: application/json" -H "Keycloak-Token: ${KEYCLOAK_TOKEN}" | python3 -m json.tool"

Example for product 2493999.

llesiak@llesiakNB:~$ curl -X DELETE https://finder.creodias.eu/api/order_item/2493999 -H "Content-Type: application/json" -H "Keycloak-Token: ${KEYCLOAK_TOKEN}" | python3 -m json.tool
{
    "id": 2493999,
    "name": "S2A_MSIL2A_20220113T194651_N0301_R099_T07LCL_20220113T214508.SAFE",
    "status": "cancelled",
    "add_date": "2022-01-18T14:25:00.387474+00:00",
    "mod_date": "2022-01-18T17:27:17.144279+00:00",
    "extra_info": "",
    "result": {
        "processed_name": "S2A_MSIL2A_20220113T194651_N0301_R099_T07LCL_20220113T214508",
        "processed_size": 153890883,
        "processing_metadata": null
    },
    "order": {
        "extra": null
    }
}

As you can see only one product has been deleted.

 

 

 

 


How to order products using Finder API?

In order to start ordering products using Finder API you may use cURL library that is mainly responsible for sending HTTP requests. By default it is preinstalled on each virtual machine.

Information

cURL parameters:
-X assign a REQUEST type to your operation
-H Content Type: declare a output format
-H Keycloak-Token: define a authorization type. For our security purpose we use KeyCloak token. A proper "How-to" procedure is located in the "Obtain a KeyCloak token" section in one part of article.

Client is able to perform one POST operation:

  • order processed products: Sentinel-2 products are available in processing level 1. By ordering processed product we send a request to obtain a  product in processing level 2.
  • order external products: Product is visible in EO Finder but is not available in our repositories yet. Its status is recognized as "Orderable".

Documentation

For comprehensive API description you can visit this page: Finder API

Ordering processed product

Syntax:

curl -X POST https://finder.creodias.eu/api/order/ -H 'Content-Type:application/json' -H "Keycloak-Token: ${KEYCLOAK_TOKEN}" \
  -d '{"priority":number,
  "order_name":"name",
  "processor": "sen2cor",
  "identifier_list":["title_without_safe_extension"]}' | python -m json.tool

Title is a representative fullname of the product in our repository. In the identifier_list parameter above you have to place title without its extension SAFE extension e.g S2A_MSIL1C_20190521T063631_N0207_R120_T40RES_20190521T094626.

Comment: You must fill in a "processor" field to accomplish an order. Its value should be set on sen2cor which is responsible for processing level 1 product to level 2 in Sentinel-2 collection.

 

Example for one product from Sentinel-2 collection:

curl -X POST https://finder.creodias.eu/api/order/ -H 'Content-Type:application/json' -H "Keycloak-Token: ${KEYCLOAK_TOKEN}" \
  -d '{"priority":1,
  "order_name":"test_order_for_FAQ",
  "processor": "sen2cor",
  "identifier_list":["S2A_MSIL1C_20190228T110001_N0207_R094_T31VEG_20190228T111324"]}' | python -m json.tool

 


Checking the order list

Syntax:

curl -X GET https://finder.creodias.eu/api/order/ -H "Content-Type: application/json" -H "Keycloak-Token: ${KEYCLOAK_TOKEN}" | python -m json.tool

 


Verifying the status of a particular order

Syntax:

curl -X GET https://finder.creodias.eu/api/order/order_id/order_items/ -H "Content-Type: application/json" -H "Keycloak-Token: ${KEYCLOAK_TOKEN}" | python -m json.tool

 

Example for order with ID 203:

curl -X GET https://finder.creodias.eu/api/order/203/order_items/ -H "Content-Type: application/json" -H "Keycloak-Token: ${KEYCLOAK_TOKEN}" | python -m json.tool

 

Attention!
If the amount of ordered products exceeed over 10, you will have to add page parameter to iterate over whole list.
One page can write only ten items to standard stream output.

e.g

curl -X GET https://finder.creodias.eu/api/order/86283/order_items/?page=2 \
-H "Content-Type: application/json" -H "Keycloak-Token: ${KEYCLOAK_TOKEN}" | python -m json.tool

You can see a total count number on every GET query in the first row:

"total": 89,
"page": 0,
"size": 10                                       

The API will no longer provide "previous" and "next" URLs. The user should iterate over the results until the number of items in the response is smaller than "size" or the following condition is met: "size" x "page" >= "total".


Obtain a KeyCloak token

IMPORTANT: Please keep in mind that the token lifespan is ten minutes. After that, you should repeat whole procedure.
export KEYCLOAK_TOKEN=$(curl -d 'client_id=CLOUDFERRO_PUBLIC' \
                             -d "username=${OS_USERNAME}" \
                             -d "password=${OS_PASSWORD}" \
                             -d 'grant_type=password' \
                             'https://auth.creodias.eu/auth/realms/DIAS/protocol/openid-connect/token' | \
                             python -m json.tool | grep "access_token" | awk -F\" '{print $4}')
Warning concerning curl statement above: Variables $OS_USERNAME and $OS_PASSWORD already assume that you have used “source” command or "." on the RC file with your OpenStack credentials. For more I refer to FAQ How to install OpenStackClient (Linux)?

Here you can locate the button to proceed download in Horizon panel:

 

You can also obtain token without using Python environment and authenticating with OpenStack RC File using simpler curl command:

curl \
   -d 'client_id=CLOUDFERRO_PUBLIC' \
   -d 'username=<your_CREODIAS_username>' \
   -d 'password=<your_CREODIAS_password>' \
   -d 'grant_type=password' \
   'https://auth.creodias.eu/auth/realms/DIAS/protocol/openid-connect/token' \
   | python -m json.tool | grep "access_token"

 

Please note that using this curl command will not save your token in environmental variables and requires entering your username and password.

Troubleshooting

In the case of failure you may encounter those symptoms:

A)

{"ErrorMessage":"orderProduct- Forbidden","ErrorCode":403}
Take a look at your Keycloak token and verify if it is correctly stored in variable:
printenv KEYCLOAK_TOKEN

B)

{"ErrorMessage":"processModuleRoute - Not Found","ErrorCode":404}

Verify your Module Path. Your request might reffer to the operation that does not exist in API installment. (Spell checking recommended)

C)

Be aware that curl parameter does not escape correctly the & as the last character in the password.
In this case you should obtain a respond:
{
    "error": "invalid_grant",
    "error_description": "Invalid user credentials"
}