TAGS

There are no tags associated with this article.

How to access/list EO DATA using boto3?

Attention!
To use boto3 your virtual machine has to be initialized in project with eo data.
We strongly recommend using virtualenv for isolating python packages. Configuration tutorial is presented below:
How to install Python virtualenv/virtualenvwrapper?

 

If virtualenv is activated:

  $ pip install boto3

Or if we install package globally:

  $ sudo pip install boto3

 

Two examples for listing products:

 

Client:

  • low-level service access
  • generated from service description
  • exposes botocore client to the developer
  • typically maps 1:1 with the service API
  • snake-cased method names (e.g. ListBuckets API => list_buckets method)
import boto3

access_key='anystring'
secret_key='anystring'

host='http://data.cloudferro.com'

s3=boto3.client('s3',aws_access_key_id=access_key,
aws_secret_access_key=secret_key, endpoint_url=host,)

for i in s3.list_objects(Delimiter='/', Bucket="DIAS", Prefix='Sentinel-2/MSI/L1C/2020/01/08/',MaxKeys=30000)['CommonPrefixes']:
        print(i)

 

Resource:

  • higher-level, object-oriented API
  • generated from resource description
  • uses identifiers and attributes
  • has actions (operations on resources)
  • exposes subresources and collections

 

import boto3

access_key='anystring'
secret_key='anystring'

host='http://data.cloudferro.com'


s3=boto3.resource('s3',aws_access_key_id=access_key,
aws_secret_access_key=secret_key, endpoint_url=host,)

bucket=s3.Bucket('DIAS')

for obj in bucket.objects.filter(Prefix='Sentinel-2/MSI/L1C/2016/02/03/S2A_OPER_PRD_MSIL1C_PDMC_20160204T043037_R085_V20160203T203520_20160203T203520.SAFE/'):
     print('{0}:{1}'.format(bucket.name, obj.key))

Save your file with .py extension and run with the "python [filename.py]" command in your terminal.

How to Add/Remove Floating IP’s to your VM?

In order to make your VM accessible from the Internet, you need to use Floating IP’s. Floating IP’s in Openstack are public IP addresses assigned to your Virtual Machine. Floating IP assignment allows you (when you have Security Groups set properly) to host services like SSH, HTTP or other over the Internet.

 

How to assign a Floating IP to your VM:

In Instances tab in Horizon, click the dropdown menu next to your VM and choose „Associate Floating IP”

 

You will be shown a window like this one:

You may choose an address from the dropdown menu, but if it's empty, you need to allocate an address first. Click "+" icon on the right.

 

There are two networks with external IP's available: external and external2. We recommend using external2 network for all your new instances. If you have virtual machines connected to the external network and want to connect them with new machines from external2 network, please use this manual How to make VMs from external and external2 communicate

external → 185.178.xxx.xxx

external2 → 43.130.xxx.xxx

Both work on different networks, which can be set up by following steps from this manual:  How to create a network with router in horizon dashboard

 

 

Select the pool you are interested in and click "Allocate IP".

 

Select newly allocated IP address and click Associate.

 

Notice: the IP address should be associated with a local address from "192.168.x.x" subnet. If you have "10.x.x.x" address change it to "192.168.x.x" address.

Click "Associate".

 

Notice: The VM's communicate between themselves trough internal network "192.168.x.x" so if you are connecting from one Virtual Machine to another you should use private addresses. If you try to connect your VM to the wrong network you will be notified by the following the message.

 

You now have public IP assigned to your instance and visible in Instances menu:

You can now connect to your Virtual Machine trough SSH or RDP from the Internet.

 

 


How to disassociate Floating IP:

If you no longer need public IP address you may disassociate it from your VM. Click "Dissasociate Floating IP" from the dropdown menu:

 

How to release Floating IP (return it to the pool)

Floating IP’s as any other Openstack resources have their cost when kept reserved and not used.

If you don't want to keep your Floating IP’s reserved for your project you may release them to the Openstack pool for other users which will also reduce your project costs.

Go to Project → Network → Floating IPs

For the address that is not in use, the "Release Floating IP" option will be available. Click it to release the IP address.


How to create a network with router in Horizon Dashboard?

When you create a new project in Horizon, its content is empty. You have to manually configure your private network. In order to complete this task, please follow those steps.

 

1. Log in to your OpenStack dashboard and choose Network tab.

2. Then choose Networks sub-label.

 

3. Click on the “Create Network” button.

4. Define your Network Name and tick two checkboxes: Enable Admin State and Create Subnet. Go to Next.

5. Define your Subnet name. Assign a valid network address with mask presented as a prefix. (This number determines how many bytes are being destined for network address)

Define Gateway IP for your Router. Normally it’s the first available address in the subnet.

Go to Next.

 

6. In Subnet Details you are able to turn on DHCP server, assign DNS servers to your network and set up basic routing.

In the end, confirm the process with “Create” button.

7. Click on the Routers tab.

 

8. Click on the “Create Router” button.

9. Name your device and assign the only available network → external. Finish by choosing “Create Router” blue button.

10. Click on your newly created Router (e.g called “Router_1”).

11. Choose Interfaces.

12. Choose + Add Interface button.

13. Assign a proper subnet and fill in IP Address. (It’s the gateway for our network). Submit the process.

14. The internal interface has been attached to the router.


How to make VMs from external and external2 communicate

This manual describes the situation when you have an instance or more instances connected to "external" typology. Every new instance connected to new "external2" typology has no network connection with old "external" typology by default. Following steps will guide you to new network creation in order to have "external" VMs and "external2" VMs connected.

If you don't have an instance connected to "external2" typology, please follow this guide: How to add floating IP to VM from external2 pool?

If you don't have an instance, please follow this guide: How to create new VM in OpenStack dashboard (Horizon)?

 

In this scenario, we use one instance connected to external network and the second instance connected to external2 network:

In order to connect those two networks, another network has to be created. 

In Project → Network → Networks choose "Create Network"

 

Provide any name for the new network, in this example, it will be called "private_LAN".

 

Provide "Subnet Name', "Network Address" and "Gateway IP". Network address has to be different than already used networks. We have 192.168.0.0/24 network used already in private_on_external and private_on_external2 networks so we use 192.168.8.0/24.

 

We need also provide allocation pools and DNS Name Servers.

 

The new network setup is compleated now. It will be visible in Project → Networks as below:

 

In Project → Compute → Instances there are two virtual machines on different external IPs

 

For both virtual machines choose "Attach Interface" in "Actions" menu

 

In "Networks" choose newly created "private_LAN"

 

 

And finish the setup by selecting "Attach Interface"

From now on both machines are connected through the new network. You can ping them by IPs from 192.168.8.X interfaces which can be checked by ifconfig command.


How to install OpenStackClient (Windows)? GitBash/Cygwin

OpenStackClient is very useful tool to gain a powerful management of our projects in Command Line Interface.It implements new features and advantages: You can run the commands from the CLI, or include prepared scripts in Python to automate the functionality of your cloud storage.Moreover we could reach our OpenStack access on any computer if we inset credentials (username and password).

Eventually everyone may admit that generally OpenStackClient provides more opportunities to look into our compute facility deeply and much more precisely.

In Microsoft Family Systems the case of implementation of serviceable CLI client is more complicated than in Linux Distributions. Configuration file for Openstack is not prepared for clear injection into CMD or Powershell and the "export" syntax is not recognized. Hence we need to install bash emulators.

To supply our operating system into Manageable OpenstackClient we have to accessorize our Windows in:

List:

  • Python 2 or 3 ( this tutorial is prepared for 2.7.8 version)
  • Pip (tool for installing python packages)
  • GIT for Windows or Cygwin64 Terminal
  • Microsoft Visual C++ Compiler for Python 2.7

Let's start with the Python Installation. Move on to official website with all stable releases:

Step one:

www.python.org

Head to Downloads and next to Windows while the list is currently enrolled.

Pick up the latest version for Python 2. In our case it is seen 2.7.15.

Go through the installation process normally. Default location for Python will be placed in your OS partition C:  such as:

C:\Python27

Step two:

To simplify our actions with Python and his dependencies we need to set up environmental variables.

Move on to our control panel:

Go straight through the main category System and Security and next click on the System tab

The left panel consists of three options. Choose Advanced system settings.

In the system properties pop-up window choose Environment Variables:

In current state we should be able to take a look at two tables

  • User variables for Administrator
  • System variables

We are interested in the blue-captured variable Path. Click the "Edit" button below three records: Path,TEMP and TMP

Add three rows to the Path:

#Python localization
C:\Python27
#virtualenv localization(we will be using this package to implement environments for python)
C:\Python27\Lib\source-packages\
#pip localization
C:\Python27\Scripts\

Confirm applied changes and close all remaining windows.

Step three:

Download and install GIT for Windows to emulate terminal. It allow to use typical and straighly-used linux commands like: ls, source, vim etc. It has the major role in our exercise in the final stage.

https://gitforwindows.org

After installation you should acquire three GIT extensions:

  • Bash
  • CMD
  • GUI

From this instance use GIT Bash till the end to run commands and .sh files.

Download the get-pip.py

https://bootstrap.pypa.io/get-pip.py

Go to the Downloads directory and execute the command:

python get-pip.py

Now we are handling a useful tool to manage and install python packages.

Occasionally we may want to check if everything is alright. Type in an example:

pip -V

If we got an output answer about pip version we might assume a half of our success.

It is strongly important to include this command. It update our PythonSSL certification which is being checked during pip package installations:

pip install -U requests[security]

Furthermore we should implement visual c++ compiler for python 2.7

https://www.microsoft.com/en-us/download/details.aspx?id=44266

Change your directory to C:\Users\Administrator and install virtual environment package

pip install virtualenv

Syntax for creating a new environment:

#virtualenv (name)
virtualenv test

A new directory should appear in our current folder

TO use test env we need to run activate file.

cd test/Scripts
source Activate

 

Environment is active.

Step four:

Install python-openstackclient in our separated test environment:

pip install python-openstackclient

Download your rc v3 file from your horizon panel. Precised instruction is presented in the "Python Openstackclient for Linux"

Now we need to add two additional lines to our file to proceed the authorization process.

vim (file_name)

The rows you need to type in are:

export OS_VOLUME_API_VERSION=2
export OS_IMAGE_API_VERSION=2

Save and source your rc file.

source (file_name)

Enter your password

Every credentials has been equipped. From the theoretical point view we should be able to enter openstack commands associated with our projects.

Type in:

openstack server list

The provisioned table in your console should appear without any bad circumstances.

If you experience HTTP 400 error then your rc file has got a problem with a variables (check if you added necessary records)

If you experience HTTP 401 error then you probably typed in wrong password. Repeat your source command.