TAGS

There are no tags associated with this article.

How can I open new ports (e.g. port 80 for http) for my service/instance?

To open new port for new service on the instance, click Project -> Network -> Security Groups and click "Create Security Group".

By default on the new group you will have two Egress (outgoing) rules, for ipv4 and ipv6.

You need to create new Ingress (incoming) rule that should look like this:

Ingress    IPv4    TCP    80 (HTTP)    0.0.0.0/0


After creating the new Security Group you have to add it to your instance.

In the instance settings click "Security Groups" and add your newly created Security Group to your instance.

 


EODATA ACCESS - S3 OR NFS

S3FS in Linux

In instances started from Linux images created after March 14,2020, /eodata is automatically mounted after adding the "eodata" network to your VM.

The /eodata mount runs as a service and can be restarted by executing the command:

sudo systemctl restart eodata.mount

and the status can be checked by executing:

sudo systemctl status eodata.mount

In instances created before March 14/2020, /eodata is mounted using the /etc/fstab file and can be remounted by executing commands below:

sudo umount -lf /eodata
sudo mount /eodata

NFS in Windows

In Windows:

Run CMD (Command Prompt) with administrator privileges.

Open START menu and type "CMD". Right click on the top result and select Run as Administrator.

Type:

route add 10.97.0.0/16 10.11.0.1

 

and confirm by pressing Enter.

 

The run script "mount eodata" from desktop.

 

NFS in Linux

If you want to mount eodata in Linux using NFS, you should stop the S3FS eodata.mount service:

sudo systemctl stop eodata.mount

Modify /etc/fstab and uncomment the line:

nfs.eodata.cloudferro.com:/eodata/repository /eodata nfs ro,noauto,_netdev 0 0

Save /etc/fstab and invoke:

sudo mount /eodata

Can't access EODATA

If you have problems with access to eodata try the following:

 

install arping:

in CentOS

sudo yum install arping

in Ubuntu:

sudo apt install arping

check the name of the interface connected to eodata network:

ifconfig

based on the response, find the number of  the interface of 10.111.x.x (eth<number> or ens<number>)

after that invoke the following commands:

in CentOS:

sudo arping -U -c 2 -I eth<number> $(ip -4 a show dev eth1 | sed -n 's/.*inet \([0-9\.]\+\).*/\1/p')


in Ubuntu:

sudo arping -U -c 2 -I ens<number> $(ip -4 a show dev ens4 | sed -n 's/.*inet \([0-9\.]\+\).*/\1/p')

 

Next ping data.cloudferro.com again. If you receive answers, remount the resource:

sudo umount -lf /eodata
sudo mount /eodata

In the instances created after March 14, 2020, /eodata is automatically mounted after adding the "eodata" Network to your VM.

The /eodata mount runs as a service and can be restarted by executing the command:

sudo systemctl restart eodata.mount

and status can be checked by executing:

sudo systemctl status eodata.mount

In the instances created before March 14, 2020, /eodata is mounted using the /etc/fstab file and can be remounted by executing commands below:

sudo umount -lf /eodata
sudo mount /eodata

 In Windows:

Run CMD (Command Prompt) with administrator privileges.

Open START menu and type "CMD". Right click on the top result and select Run as Administrator.

Type:

route add 10.97.0.0/16 10.11.0.1

 and confirm by pressing Enter.

The run script "mount_eodata" from desktop.

 

 


How to generate KeyCloak token?

a) Using web browser console

1. Open your web browser and enter https://finder.creodias.eu/www/ in the address field.

 

2. Click on “Log In” button located in the upper-right-corner.

 

3. Fill in the blanks with your e-mail address and password.

 

4. Press “Login”.

 

5. Click somewhere on the site with the right mouse button and choose “Inspect Element”.

 

6. Change your view side from Inspector to Console.

 

 

7. Click on the white space beside the blue arrow to start typing.

 

8. Type in keycloak.token and press enter in order to generate your token.


b) By query with cURL

This case has been presented in the How to order products using Finder API already. CURL is a tool to send data to the server using several protocols such as HTTP. In this example, the output is being filtred by grep and awk commands to obtain a token. In the Linux operating system it's being seen as environmental variable KEYCLOAK_TOKEN.

export KEYCLOAK_TOKEN=$(curl -d 'client_id=CLOUDFERRO_PUBLIC' \
                             -d "username=${OS_USERNAME}" \
                             -d "password=${OS_PASSWORD}" \
                             -d 'grant_type=password' \
                             'https://auth.creodias.eu/auth/realms/DIAS/protocol/openid-connect/token' | \
                             python -m json.tool | grep "access_token" | awk -F\" '{print $4}')

c) By Python script

import json
import requests

def get_keycloak(username: str, password: str) -> str:
    data = {
        "client_id": "CLOUDFERRO_PUBLIC",
        "username": username,
        "password": password,
        "grant_type": "password",
    }
    try:
        r = requests.post(
            "https://auth.creodias.eu/auth/realms/DIAS/protocol/openid-connect/token",
            data=data,
        )
        r.raise_for_status()
    except Exception as e:
        raise Exception(
            f"Keycloak token creation failed. Reponse from the server was: {r.json()}"
        )
    return r.json()["access_token"]

keycloak_token = get_keycloak(username, password)

 


Recommended usage of access interfaces to EODATA

There are four data access interfaces used for accessing and downloading products from EO Data repository:

  • S3FS - emulated file based access over S3 interface 
  • NFS - file based access
  • S3 - object storage based interface - the recommended data access interface in CREODIAS
  • Download - zipped products from https://finder.creodias.eu

Apart from the way of data access, these interfaces differ in data transfer speed and reliability.

S3FS

While S3FS is a stable file based access to EO Data repository, it is not so fast. You may expect transfers of about 30 MB/s. Please note that S3FS does not work in Windows virtual instances since there is no S3FS client for Windows. There are some commercial alternatives such as https://tntdrive.com/ or https://www.cloudberrylab.com/drive.aspx. These alternatives are neither provided in CREODIAS Windows images nor covered by CREODIAS support.

With S3FS you may expect throughput of c.a. 30 MB/s.

S3FS is accessible only from virtual instances in CloudFerro clouds.

S3FS is the default file-based access in CREODIAS Linux instances.

NFS

NFS is much faster than S3FS file based access to EO Data, but not as stable, therefore sometimes one may experience "Stale file handle" or "Permission denied" errors. In such case it is recommended to remount the share.

With NFS you may expect at least 80-120 MB/s and more. A failure rate of ~0,1% in access to data should be handled on the application level. In case f a failure it is reFor instance remount the share or retry access to data instead of failing the application.

NFS is accessible only from virtual instances in CloudFerro clouds.

NFS access is available by default for Windows instances.

S3

If you have a possibility to access EO Data directly with S3 object storage interface from your application, you might use boto3 library in Python scripts (How to download EO Data file using boto3) or other S3 libraries depending on your programming language. It is the fastest and most efficient way to access EO Data.

S3 provides throughput up to 200 MB/s for applications over API access or via s3cmd command line tool. (How to access EO DATA and Object Storage using s3cmd (Linux))

S3 interface is accessible only from virtual instances in CloudFerro clouds.

S3 is the recommended data access interface for EO Data in CREODIAS.

Download

Users who don't have resources in CloudFerro clouds, may use CREODIAS Finder (https://finder.creodias.eu) to download complete EO products from EO Data repository in form of a single ZIP file.

The expected throughput of the download interface is about 25 MB/s.

Products downloading is available for every registered CREODIAS user.