How to access EO DATA and Object Storage using s3cmd (Linux)

How to access EO DATA using s3cmd (Linux)

Attention!
Your virtual machine has to be launched in project with EO DATA!

You can install the s3cmd using Python PIP or from Linux repository.

Installation from system repository on Debian/Ubuntu systems:

Check for updates:
 

$ sudo apt update


Installing from repository:
 

$ sudo apt install s3cmd

 

Installation from Python repository (on most Linux distributions with python and pip preinstalled):

Installing with PIP:

Check if you have PIP installed

$ pip

The program 'pip' is currently not installed. To run 'pip' please ask your administrator to install the package 'python-pip'

If not installed (Ubuntu):

$ sudo apt install python-pip

$ pip --version

pip 8.1.1 from /usr/lib/python2.7/dist-packages (python 2.7)

$ sudo pip install s3cmd

If you see the following:

Traceback (most recent call last):

File "/usr/bin/pip", line 11, in <module>

sys.exit(main())

File "/usr/lib/python2.7/dist-packages/pip/init.py", line 215, in main

locale.setlocale(locale.LC_ALL, '')

File "/usr/lib/python2.7/locale.py", line 581, in setlocale

return _setlocale(category, locale)

locale.Error: unsupported locale setting


add the following line:

export LC_ALL=en_US.UTF-8

to the file:

~/.profile

Now you can check the .profile:

$ cat ~/.profile

export LC_ALL=en_US.UTF-8

$ source ~/.profile

$ s3cmd --version

s3cmd version 2.0.1
 


Configure s3cmd
 

$ s3cmd --configure


Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.

Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables.
 

Access Key [access]:<ENTER>

Secret Key [access]:<ENTER>


Default Region [RegionOne]: <ENTER>


Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3.

S3 Endpoint [data.cloudferro.com:] <ENTER>


Use "%(bucket)s.s3.amazonaws.com" to the target Amazon S3. "%(bucket)s" and "%(location)s" vars can be used


if the target S3 system supports dns based buckets.

DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]:  <ENTER>


Encryption password is used to protect your files from reading

by unauthorized persons while in transfer to S3

Encryption password: <ENTER>

Path to GPG program [/usr/bin/gpg]: <ENTER>


When using secure HTTPS protocol all communication with Amazon S3

servers is protected from 3rd party eavesdropping. This method is

slower than plain HTTP, and can only be proxied with Python 2.7 or newer

Use HTTPS protocol [No]: <ENTER>


On some networks all internet access must go through a HTTP proxy.

Try setting it here if you can't connect to S3 directly

HTTP Proxy server name: <ENTER>


New settings:

   Access Key: access

   Secret Key: access

   Default Region: RegionOne

   S3 Endpoint: data.cloudferro.com

   DNS-style bucket+hostname:port template for accessing a bucket: %(bucket)s.s3.amazonaws.com

   Encryption password:

   Path to GPG program: /usr/bin/gpg

   Use HTTPS protocol: False

   HTTP Proxy server name: _____

   HTTP Proxy server port: 0


 Test access with supplied credentials? [Y/n] <ENTER>

 Please wait, attempting to list all buckets...

 Success. Your access key and secret key worked fine :-)


 Now verifying that encryption works...

 Not configured. Never mind.


 Save settings? [y/N]  y <ENTER>

 Configuration saved to '/home/eouser/.s3cfg'


Now you can use s3cmd commands (additional information about s3cmd: http://s3tools.org/usage)
 

$ s3cmd ls

2017-12-11 15:30  s3://DIAS

2017-12-11 15:30  s3://EOCLOUD

2017-12-11 15:30  s3://EODATA


$ s3cmd ls s3://EODATA/

                       DIR   s3://EODATA/Envisat/

                       DIR   s3://EODATA/Landsat-5/

                       DIR   s3://EODATA/Landsat-7/

                       DIR   s3://EODATA/Landsat-8/

                       DIR   s3://EODATA/Sentinel-1/

                       DIR   s3://EODATA/Sentinel-2/

                       DIR   s3://EODATA/Sentinel-3/

                       DIR   s3://EODATA/Sentinel-5P/

 

In order to acquire access to Object Storage buckets via s3cmd, first you have to generate your own ec2 credentials with this tutorial.
After creation of credentials please remove file .s3cfg in Home folder and then reconfigure s3cmd by entering:

s3cmd --configure 

and following values:

New settings:
Access Key: (your EC2 credentials)
Secret Key: (your EC2 credentials)
Default Region: none
S3 Endpoint: cf2.cloudferro.com:8080
DNS-style bucket+hostname:port template for accessing a bucket: cf2.cloudferro.com:8080
Encryption password: (your password)
Path to GPG program: /usr/bin/gpg
Use HTTPS protocol: True
HTTP Proxy server name:
HTTP Proxy server port: 0

After this operation, you should be allowed to list and access your Object Storage.


How to download CREODIAS products using EO Finder?

 

First, it is required to log in on CREODIAS in order to download satellite products. Click “Log In” button

 

 

You will be moved to login site and asked to enter your login and password.

 

 

 

Click “Data & Analytics” and choose EOFinder

 

 

Select desired product (by entering REST query or finding it by in-built search engine, as explained here

 

 

Click on product path in “Search Results” panel

 

 

Choose “Download” icon

 

 

Now you will be able to download product to your computer.

 

 

As well as using graphic interface of EOFinder, you can browse and download satellite products by s3cmd, s3fs and boto3 using terminal:

How to mount EO DATA using s3fs?
How to mount EO DATA using s3 protocol?

How to access EO DATA using s3cmd?
How to download EO DATA file using boto3?


How to download EOData products using EOFinder API?

Example of acquiring EOData product using zipper service.

1. Enter the CREODIAS Finder web page.

https://finder.creodias.eu/www/

2. If you want to download EOData product, please search for the desired product by providing sufficient parameters in the form. After that, verify the results and locate the REST query that has been generated.

a) Using filters in order to find a desired product:

b) Checking the results and locating REST query:

3. Copy and paste it in the we browser address bar. Run this query.

4. Press Ctrl + F in order to find a "download" paragraph. Look at the prepared zipper link and copy it.

5. Add "?token=" sequence at the end of the URL.

https://zipper.creodias.eu/download/db0c8ef3-8ec0-5185-a537-812dad3c58f8?token=

6. Generate your keycloak token (remember to source your OpenRC file first)

export KEYCLOAK_TOKEN=$(curl -d 'client_id=CLOUDFERRO_PUBLIC' \
                             -d "username=${OS_USERNAME}" \
                             -d "password=${OS_PASSWORD}" \
                             -d 'grant_type=password' \
                             'https://auth.creodias.eu/auth/realms/DIAS/protocol/openid-connect/token' | \
                             python -m json.tool | grep "access_token" | awk -F\" '{print $4}')

7. Execute wget command:

wget -cO - https://zipper.creodias.eu/download/db0c8ef3-8ec0-5185-a537-812dad3c58f8?token=(paste token here) > myproduct.zip

8. After finished operation make sure that the product has been downloaded properly.

2019-02-18 16:21:08 (37,7 MB/s) - written to stdout [432095955]


[eouser@CREODIAS_MACHINE ~]$ ll | grep myproduct.zip

-rw-rw-r-- 1 eouser eouser 432095955 lut 18 16:21 myproduct.zip

[eouser@CREODIAS_MACHINE ~]

How to activate Windows RDP Licensing in my VM?

In case you need more than one parallel users access to the Windows server via Remote Desktop Protocol, you have to purchase Remote Desktop License for each new user.

Windows RDP Client is licensed per user.

If you want to activate RDP licensing in your domain, please send us a support request (here you can see how to do it) or contact our Sales Department at sales@creodias.eu


How to order products using Finder API?

In order to start ordering products using Finder API you may use cURL library that is mainly responsible for sending HTTP requests. By default it is preinstalled on each virtual machine.

Information

cURL parameters:
-X assign a REQUEST type to your operation
-H Content Type: declare a output format
-H Keycloak-Token: define a authorization type. For our security purpose we use KeyCloak token. A proper "How-to" procedure is located in the "Obtain a KeyCloak token" section in one part of article.

Client is able to perform one POST operation:

  • order processed products: Sentinel-2 products are available in processing level 1. By ordering processed product we send a request to obtain a  product in processing level 2.

In the future we will also add another functionality:

  • order external products: Product is visible in EO Finder but is not available in our repositories yet. Its status is recognized as "Orderable".

Ordering processed product

Syntax:

curl -X POST https://finder.creodias.eu/api/order/ -H 'Content-Type:application/json' -H "Keycloak-Token: ${KEYCLOAK_TOKEN}" \
  -d '{"priority":number,
  "order name":"name",
  "processor": "sen2cor",
  "identifier_list":["Product identifier"]}' | python -m json.tool

 

Comment: You must fill in a "processor" field to accomplish an order. Its value should be set on sen2cor which is responsible for processing level 1 product to level 2 in Sentinel-2 collection.

 

Example for one product from Sentinel-2 collection:

curl -X POST https://finder.creodias.eu/api/order/ -H 'Content-Type:application/json' -H "Keycloak-Token: ${KEYCLOAK_TOKEN}" \
  -d '{"priority":1,
  "order name":"test_order_for_FAQ",
  "processor": "sen2cor",
  "identifier_list":["S2A_MSIL1C_20190228T110001_N0207_R094_T31VEG_20190228T111324"]}' | python -m json.tool

 


Checking the order list

Syntax:

curl -X GET https://finder.creodias.eu/api/order/ -H "Content-Type: application/json" -H "Keycloak-Token: ${KEYCLOAK_TOKEN}" | python -m json.tool

 


Verifying the status of a particular order

Syntax:

curl -X GET https://finder.creodias.eu/api/order/order_id/ -H "Content-Type: application/json" -H "Keycloak-Token: ${KEYCLOAK_TOKEN}" | python -m json.tool

 

Example for order with ID 203:

curl -X GET https://finder.creodias.eu/api/order/203/ -H "Content-Type: application/json" -H "Keycloak-Token: ${KEYCLOAK_TOKEN}" | python -m json.tool

 


Obtain a KeyCloak token

IMPORTANT: Please keep in mind that the token lifespan is ten minutes. After that, you should repeat whole procedure.

export KEYCLOAK_TOKEN=$(curl -d 'client_id=CLOUDFERRO_PUBLIC' \
                             -d "username=${OS_USERNAME}" \
                             -d "password=${OS_PASSWORD}" \
                             -d 'grant_type=password' \
                             'https://auth.creodias.eu/auth/realms/DIAS/protocol/openid-connect/token' | \
                             python -m json.tool | grep "access_token" | awk -F\" '{print $4}')



Warning concerning curl statement above: Variables $OS_USERNAME and $OS_PASSWORD already assume that you have used “source” command or "." on the RC file with your OpenStack credentials. For more i reffer to FAQ "How to install OpenStackClient (Linux)?" LINK

Here you can locate the button to proceed download in Horizon panel:

 

You can also obtain token without using Python environment and authenticating with OpenStack RC File using simpler curl command:

curl \
   -d 'client_id=CLOUDFERRO_PUBLIC' \
   -d 'username=<your_CREODIAS_username>' \
   -d 'password=<your_CREODIAS_password>' \
   -d 'grant_type=password' \
   'https://auth.creodias.eu/auth/realms/DIAS/protocol/openid-connect/token' \
   | python -m json.tool | grep "access_token"

 

Please note that using this curl command will not save your token in environmental variables and requires entering your username and password.

Troubleshooting

In the case of failure you may encounter those symptoms:

A)

{"ErrorMessage":"orderProduct- Forbidden","ErrorCode":403}
 
printenv KEYCLOAK_TOKEN

Take a look at your Keycloak token and verify if it is correctly stored in variable:

B)

{"ErrorMessage":"processModuleRoute - Not Found","ErrorCode":404}

Verify your Module Path. Your request might reffer to the operation that does not exist in API installment. (Spell checking recommended)

C)
Be aware that curl parameter does not escape correctly with the & as the last sign in the password.
In this case you should obtain a respond:

{
    "error": "invalid_grant",
    "error_description": "Invalid user credentials"
}