How to install OpenStackClient (Linux)?
OpenStackClient is very useful tool to gain a powerful management of our projects in Command Line Interface. It implements new features and advantages: You can run the commands from the CLI, or include prepared scripts in Python to automate the functionality of your cloud storage. Moreover we could reach our OpenStack access on any computer if we inset credentials (username and password).
Eventually everyone may admit that generally OpenStackClient provides more opportunities to look into our compute facility deeply and much more precisely.
Attention: It is strongly recommended to use virtual environments which do not affect system variables globally. If something goes wrong, everything is happening in separated space. |
FAQ covers installation of Openstack Client under Ubuntu 18.04 LTS and Python 3. This method works on Ubuntu 20.04 as well.
Installation under other distributions may be simillar.
Let's start with the apt update && apt upgrade commands. This will update our system packages to the latest version
sudo apt update && sudo apt upgrade
Now let's install python3 and create a new python enviroment called simply "openstack_cli":
sudo apt install python3-venv python3 -m venv openstack_cli source openstack_cli/bin/activate
Now upgrade Python Package Installer (pip) to the latest version using command below:
pip install --upgrade pip
Finally install python-openstackclient:
pip install python-openstackclient
After this, you should be able to run openstack command from consone, eg:
openstack --help
If everything seems to work, we might firmly move on to our Horizon Panel.
Log in to your account.
Head straight to our email button in the upper-right-corner. Click on it.
Choose Openstack RC File v2 or v3.
Save your RC file on your disk.
If you use some text editor like vim it should consist of many variables related to your domain account like it:
Change your directory to downloading destination and execute the configuration file:
For example t will look like: cloud_xxxxx\project_without_eo-openrc.sh
Time to run this sh file:
. cloud_xxxxx\project_without_eo-openrc.sh |
The prompted message will ask you for your domain password. Type it in and press enter.
If the password is correct then you've got access granted.
After that you can eg. check list of your servers by typing in console:
openstack server list
Output of this command should contain table with ID, Name, Status, Networks, Image and Flavor of your virtual machines.
We recommend checking Openstack documentation which contains lists of available commands. There you may reconsider many scenarios linked with your management.
v.2020-05-19
How to use JupyterHub?
First of all, to use JupyterHub you have to be registered and signed in Creodias.
Please see: How to register to Creodias Portal?
After that, choose Tools → JupyterHub from navigation menu:
Then you can see main page of JupyterHub. It is highly recommended to create new projects in persistent directory to avoid data loss caused by e.g. by restart of your Jupyter Notebook container.
You can create a new script by choosing New and the language of your choice e.g. Python 3.
Product Query and selection
You can define search parameters and query the CREODIAS Finder engine for products that correspond to chosen parameters. Finder API is describerd here:
https://creodias.eu/eo-data-finder-api-manual
To run the script you can use "Run" button on the top or use shortcut "Ctrl + Enter".
All shortcuts are available after clicking keyboard icon, there you can also define your own shortcuts. Keep in mind that you have to save your script.
Below you can find the above code.
Query = 'Tokio%20Winter' # here you can define semantic search Cloud = '[0,5]' # here you can define cloud cover in percentage sort = 'sortParam=cloudCover&sortOrder=ascending' Coll = 'Sentinel2' # here you can choose which satellite are you interested in ResultNumber = 0 OutputFileName = 'TrueColor' URL = 'http://finder.creodias.eu/resto/api/collections/' + Coll +'/search.json?_pretty=true&q=' + Query + '&' + 'cloudCover=' + Cloud + '&' + sort import json, requests, os products = json.loads(requests.get(URL).text)['features'] for product in products: print(product['properties']['productIdentifier']) file = products[ResultNumber]['properties']['productIdentifier']
Product selection
Using selected product identifier (stored in "file" variable), you can open access bands to be later saved as a GeoTif.
import rasterio path = file + '/GRANULE' + '/' + os.listdir(file + '/GRANULE')[0] + '/IMG_DATA' if (os.path.isdir(path + '/' + sorted(os.listdir(path))[0])): path = path + '/' + sorted(os.listdir(path))[0] band2 = rasterio.open(path + '/' + sorted(os.listdir(path))[1]) #blue band3 = rasterio.open(path + '/' + sorted(os.listdir(path))[2]) #green band4 = rasterio.open(path + '/' + sorted(os.listdir(path))[3]) #red path
GeoTif creation
Now you can create a GeoTif file. Bands are stored in GeoTiff format and saved to the current directory of your Notebook.
We recommend storing the results in "persistent" directory.
from rasterio import plot import matplotlib.pyplot as plt trueColor = rasterio.open( OutputFileName + '.tiff','w',driver='Gtiff', width=band4.width, height=band4.height, count=3, crs=band4.crs, transform=band4.transform, dtype=band4.dtypes[0] ) trueColor.write(band2.read(1), 3) #blue trueColor.write(band3.read(1), 2) #green trueColor.write(band4.read(1), 1) #red trueColor.close() os.path.isfile(OutputFileName + '.tiff')
Displaying the imageNow you can display the image.
|
How to search for satellite products using tiles?
In CREODIAS Finder you can search Sentinel-2 products by date and tile, rather than by geometry.
What exactly is a tile?
For Level-1C and Level-2A tiles are 100x100 km2 ortho-images in UTM/WGS84 projection.
Tile names correspond to UTM zone grid. The world is divided into 60 UTM zones of 8 degrees width in longitude, with numbers increasing towards the East. Tile names are included into products Identifiers, that allows fast searching for them.
To find an area of your interest, you can use Sentinel-2 tile grid KML file from ESA, that provides the footprint of all the tiles, and their names. This file is available under this link: Sentinel-2 tiles .kml file.
KML file can be easily opened in Google Earth Engine: https://earth.google.com/web/ as a new project.
Step by step guide is described here: Import KML map data into Google Earth.
After that select the tile you're interested in and copy it's name.
For purposes of this guide we will use tile 33UYR, illustrated above (screenshot of Google Earth Engine) and below (screenshot of CREODIAS Finder).
Searching for products using FINDER.
The easiest way to reach products from tile is to copy the tile name into search criteria and confirm by 'Enter' key.
Here are all the results for the example tile:
To narrow down the total number of products, we can use a time frame filter, for example by inserting one month period to 'observed' date.
Searching for products using Rest query.
You can use "Rest query" field to modify your search:
To do that, copy/modify/paste the string from the field:
https://finder.creodias.eu/resto/api/collections/Sentinel2/search.json?maxRecords=10&productIdentifier=%2533UYR%25&sortParam=startDate&sortOrder=descending&status=all&dataset=ESA-DATASET
Tile name in our query is specified by the following keyword:
productIdentifier=%2533UYR%25
You can also specify time period for search, by adding keywords 'startDate' and 'completionDate'.
For example add to your query, 'startDate=2022-01-17T00:00:00Z' and 'completionDate=2022-01-17T23:59:59Z', if you are searching for one day only.
https://finder.creodias.eu/resto/api/collections/Sentinel2/search.json?maxRecords=10&sortParam=startDate&sortOrder=descending&status=all&dataset=ESA-DATASET&productIdentifier=%2533UYR%25&startDate=2022-01-17T00:00:00Z&completionDate=2022-01-17T23:59:59Z
And the results of query:
Resources:
Download Sentinel-2 tile grid KML file:
Articles you may find helpful:
https://labo.obs-mip.fr/multitemp/the-sentinel-2-tiles-how-they-work/
https://sentinels.copernicus.eu/web/sentinel/user-guides/sentinel-2-msi/naming-convention
https://support.google.com/earth/answer/7365595?hl=en&co=GENIE.Platform%3DDesktop
How to delete Finder Order / Order Product via API
How to delete Order / Order Product
Using Finder API you can delete an entire order or just individual products.
To do that please make sure that you have Openstack CLI installed on your operating system. You have to also obtain KeyCloak.
How to install Openstack Client on Linux
How to install Openstack Client on Windows
How to obtain KeyCloak Token / Order products using API
Deleting an entire order
To delete an entire order you can use following command: "curl -X DELETE https://finder.creodias.eu/api/order/*order_ID*/ -H "Content-Type: application/json" -H "Keycloak-Token: ${KEYCLOAK_TOKEN}" | python3 -m json.tool"
Example for Order 1319047
llesiak@llesiakNB:~$ curl -X DELETE https://finder.creodias.eu/api/order/1319047/ -H "Content-Type: application/json" -H "Keycloak-Token: ${KEYCLOAK_TOKEN}" | python3 -m json.tool
As you can see the whole order has been deleted (canceled).
Deleting Order Products
To delete an product you'll need product item ID.
To get ID of Item that you've ordered please use this request: "curl -X GET https://finder.creodias.eu/api/order/*order_ID*/order_items -H "Content-Type: application/json" -H "Keycloak-Token: ${KEYCLOAK_TOKEN}" | python3 -m json.tool"
As a result you'll get details of an order. Example for order 1317258.
llesiak@llesiakNB:~$ curl -X GET https://finder.creodias.eu/api/order/1317258/order_items -H "Content-Type: application/json" -H "Keycloak-Token: ${KEYCLOAK_TOKEN}" | python3 -m json.tool { "items": [ { "id": 2491216, "name": "S2A_MSIL1C_20190228T110001_N0207_R094_T31VEG_20190228T111324", "status": "done", "add_date": "2022-01-14T14:05:59.329727+00:00", "mod_date": "2022-01-14T14:06:05.849253+00:00", "extra_info": "", "result": { "processed_name": "S2A_MSIL2A_20190228T110001_N0211_R094_T31VEG_20190228T113827", "processed_size": 673655801, "processing_metadata": {} }, "order": { "extra": null } } ], "total": 1, "page": 0, "size": 10 }
After getting product ID you can delete it using this command: "curl -X DELETE https://finder.creodias.eu/api/order_item/*order_Item_ID*-H "Content-Type: application/json" -H "Keycloak-Token: ${KEYCLOAK_TOKEN}" | python3 -m json.tool"
Example for product 2493999.
llesiak@llesiakNB:~$ curl -X DELETE https://finder.creodias.eu/api/order_item/2493999 -H "Content-Type: application/json" -H "Keycloak-Token: ${KEYCLOAK_TOKEN}" | python3 -m json.tool { "id": 2493999, "name": "S2A_MSIL2A_20220113T194651_N0301_R099_T07LCL_20220113T214508.SAFE", "status": "cancelled", "add_date": "2022-01-18T14:25:00.387474+00:00", "mod_date": "2022-01-18T17:27:17.144279+00:00", "extra_info": "", "result": { "processed_name": "S2A_MSIL2A_20220113T194651_N0301_R099_T07LCL_20220113T214508", "processed_size": 153890883, "processing_metadata": null }, "order": { "extra": null } }
As you can see only one product has been deleted.
How to order products using Finder API?
In order to start ordering products using Finder API you may use cURL library that is mainly responsible for sending HTTP requests. By default it is preinstalled on each virtual machine.
Information
-X assign a REQUEST type to your operation
-H Content Type: declare a output format
-H Keycloak-Token: define a authorization type. For our security purpose we use KeyCloak token. A proper "How-to" procedure is located in the "Obtain a KeyCloak token" section in one part of article.
Client is able to perform one POST operation:
- order processed products: Sentinel-2 products are available in processing level 1. By ordering processed product we send a request to obtain a product in processing level 2.
- order external products: Product is visible in EO Finder but is not available in our repositories yet. Its status is recognized as "Orderable".
Documentation
For comprehensive API description you can visit this page: Finder API
Ordering processed product
Syntax:
curl -X POST https://finder.creodias.eu/api/order/ -H 'Content-Type:application/json' -H "Keycloak-Token: ${KEYCLOAK_TOKEN}" \ -d '{"priority":number, "order_name":"name", "processor": "sen2cor", "identifier_list":["title_without_safe_extension"]}' | python -m json.tool |
Title is a representative fullname of the product in our repository. In the identifier_list parameter above you have to place title without its extension SAFE extension e.g S2A_MSIL1C_20190521T063631_N0207_R120_T40RES_20190521T094626.
Example for one product from Sentinel-2 collection:
curl -X POST https://finder.creodias.eu/api/order/ -H 'Content-Type:application/json' -H "Keycloak-Token: ${KEYCLOAK_TOKEN}" \ -d '{"priority":1, "order_name":"test_order_for_FAQ", "processor": "sen2cor", "identifier_list":["S2A_MSIL1C_20190228T110001_N0207_R094_T31VEG_20190228T111324"]}' | python -m json.tool |
Checking the order list
Syntax:
curl -X GET https://finder.creodias.eu/api/order/ -H "Content-Type: application/json" -H "Keycloak-Token: ${KEYCLOAK_TOKEN}" | python -m json.tool |
Verifying the status of a particular order
Syntax:
curl -X GET https://finder.creodias.eu/api/order/order_id/order_items/ -H "Content-Type: application/json" -H "Keycloak-Token: ${KEYCLOAK_TOKEN}" | python -m json.tool |
Example for order with ID 203:
curl -X GET https://finder.creodias.eu/api/order/203/order_items/ -H "Content-Type: application/json" -H "Keycloak-Token: ${KEYCLOAK_TOKEN}" | python -m json.tool |
If the amount of ordered products exceeed over 10, you will have to add page parameter to iterate over whole list.
One page can write only ten items to standard stream output.
e.g
curl -X GET https://finder.creodias.eu/api/order/86283/order_items/?page=2 \ -H "Content-Type: application/json" -H "Keycloak-Token: ${KEYCLOAK_TOKEN}" | python -m json.tool
You can see a total count number on every GET query in the first row:
"total": 89, "page": 0, "size": 10
The API will no longer provide "previous" and "next" URLs. The user should iterate over the results until the number of items in the response is smaller than "size" or the following condition is met: "size" x "page" >= "total".
Obtain a KeyCloak token
export KEYCLOAK_TOKEN=$(curl -d 'client_id=CLOUDFERRO_PUBLIC' \ -d "username=${OS_USERNAME}" \ -d "password=${OS_PASSWORD}" \ -d 'grant_type=password' \ 'https://auth.creodias.eu/auth/realms/DIAS/protocol/openid-connect/token' | \ python -m json.tool | grep "access_token" | awk -F\" '{print $4}') |
Here you can locate the button to proceed download in Horizon panel:
You can also obtain token without using Python environment and authenticating with OpenStack RC File using simpler curl command:
curl \ -d 'client_id=CLOUDFERRO_PUBLIC' \ -d 'username=<your_CREODIAS_username>' \ -d 'password=<your_CREODIAS_password>' \ -d 'grant_type=password' \ 'https://auth.creodias.eu/auth/realms/DIAS/protocol/openid-connect/token' \ | python -m json.tool | grep "access_token"
Please note that using this curl command will not save your token in environmental variables and requires entering your username and password.
Troubleshooting
In the case of failure you may encounter those symptoms:
A)
{"ErrorMessage":"orderProduct- Forbidden","ErrorCode":403} |
printenv KEYCLOAK_TOKEN B) {"ErrorMessage":"processModuleRoute - Not Found","ErrorCode":404} Verify your Module Path. Your request might reffer to the operation that does not exist in API installment. (Spell checking recommended) C) Be aware that curl parameter does not escape correctly the & as the last character in the password. In this case you should obtain a respond:
|