TAGS

There are no tags associated with this article.

Is it possible to make a data volume visible by several VMs simultaneously?

 

It is not possible on a cloud level of management. It is, however, possible (and widely practised) to attach such volume to one of the VM-s and export it to others by means of networking file system, e.g. NFS, supported on operating system level. See How to export volume over NFS?  . One can also use Object storage to make data available to several VMs.


How To Mount Object Storage Container as File System on Windows VM on Creodias

Note

This article assumes that you have access to CloudFerro WAW3-1 infrastructure, which has Kubernetes support built-in (OpenStack Magnum module).

If your CREODIAS account has access only to CF2 infrastructure, please contact support to get access to WAW3-1.

This article covers configuring automatic mounting of object storage containers on Windows virtual machines running on Creodias WAW3-1 cloud. Your object storage containers will be automatically mounted to your Windows VM and you will be able to access them from This PC window of the Administrator user on that VM.

What We Are Going To Cover

  • Entering the connection data

  • Performing a test mount

  • Finding the appropriate time to cache directory entries for

  • Setting automatic mounting for a container

  • Disabling the automatic mounting of a container

Prerequisites

No. 1 Account

You need a Creodias hosting account with access to the Horizon interface: https://horizon.cloudferro.com/auth/login/?next=/.

No. 2. Object storage container

You need at least one object storage container on the Creodias WAW3-1 cloud. If you do not have one yet, please follow this article: How to use Object Storage on Creodias

No. 3. Generated EC2 Credentials

You need to generate EC2 credentials for your account. If you haven’t done that yet, please follow this article: How to Generate EC2 Credentials on Creodias.

You will need to use the OpenStack CLI client to do that. The article linked above covers its installation on Linux. If you are a Windows user, please follow one of the articles linked below to install the OpenStack CLI client on Windows:

Then, adjust the instructions from the article How to Generate EC2 Credentials on Creodias accordingly.

No. 4 Virtual machine with Windows

You need a Windows VM on Creodias WAW3-1 cloud with Rclone, WinFSP and NSSM configured.

There are three methods of fulfilling this prerequisite:

Method 1: Create a new Windows virtual machine

Currently available Windows images have appropriate software preinstalled.

Use article How to create new Linux VM in OpenStack Dashboard Horizon on Creodias as the basic blueprint for creation of an instance.

Apply the following changes to the procedure in that article:

For Source, use a Windows image.

For Networks, be sure to add two networks, one starting with cloud_ and the other starting with eodata_.

If you do not want to access your instance outside of the Horizon dashboard, you may omit the following from the workflow described in that article:

  • creation of a floating IP,

  • using the allow_ping_ssh_icmp_rdp security group as well as

  • the instructions regarding the SSH connection.

The instructions regarding SSH connection from that article do not apply to Windows VMs. Virtual machines with Windows are typically controlled using RDP. If you choose this method, you might want to consider using bastion host forwarding to secure your connection:

Connecting to a Windows VM via RDP through a Linux bastion host port forwarding on Creodias

Method 2: Use a virtual machine created using an image published on or after 20th of December 2022

If you already have a virtual machine created using an image published on or after 20th of December 2022, you should have appropriate software installed (unless a user later removed it). Therefore, you should be able to use one such machine for this article.

Method 3: Use a machine created using an older image and install required software manually

If you have a virtual machine created using an image published before 20th of December 2022, you will need to install and configure appropriate software manually. Information on how to do it can be found in Method 1 of this article: How to mount eodata on Windows virtual machine on Creodias hosting. If you do not want to have access to the EODATA repository on your virtual machine, you can finish following that article after having created Rclone configuration file in the Mounting EODATA section of that text (you do not need to add any content to it).

Software tools used in this article: Rclone, WinFSP, and NSSM

All the software used in this article comes preinstalled on Creodias WAW3-1 virtual machines created using default Windows images.

Rclone has multiple functions such as managing files in cloud storage and syncing between file systems. In this article, you will use its rclone mount command to mount object storage on your Windows VM.

WinFSP enables accessing custom file systems on Microsoft Windows. In this workflow, it will allow Rclone to mount the S3 storage.

NSSM is a service manager. Here, it will be used for configuring automatic mounting of object storage. You will enter its GUI it from the command line.

How the Rclone configuration file will be used in this article

Virtual machines created using default Windows running on Creodias WAW3-1 cloud have automatic mounting of the EODATA repository configured. This process is done using a script which creates the appropriate configuration file if it doesn’t exist and mounts the repository.

In this article, you will add the appropriate login credentials for your object storage containers to that configuration file. After that, using provided program called NSSM, you will create services which will automatically mount those object storage containers.

In the end, less than a minute after each login you should see the EODATA repository and your configured object storage containers ready to use in your This PC window.

Step 1: Enter the connection data

Login to the Administrator account on your virtual machine.

Navigate to the C:\Users\Administrator\.config\rclone folder using the Windows file manager. Open the file rclone.conf in that folder using Notepad or other plain text editor like Notepad++. If you do not see that file there, wait up to a couple of minutes and try again.

The file should already contain section used for accessing the EODATA repository:

If you did not configure anything there yet, it will be empty.

Each section containing the object storage connection data starts with a line containing its name written in square brackets. In this case, such section will be used for connecting to all object storage containers stored in the same place using the same pair of EC2 credentials. If you intend to use object storage containers which have different credentials, each key pair will, however, need its own section similar to the one below.

Add the following section to the end of this file:

[remote-config]
type = s3
provider = Other
access_key_id = 1234
secret_access_key = 4321
endpoint = s3.waw3-1.cloudferro.com

In the above block, replace 1234 and 4321 with the access and secret key you obtained while following Prerequisite No. 3, respectively.

If you want to use a different name for your connection than remote-config, replace it in the code above. This name does not have to be the same as the name of one of your containers.

If you want to use object storage containers from more than one key pair, create a separate section for each of them. Each section has to have a different name written in square brackets.

As stated previously, you do not need multiple sections for different object storage containers using the same key pair.

Save the file and close Notepad.

Step 2: Perform a test mount

You can now test the connection to your object storage container. Open PowerShell and navigate to the folder containing Rclone by executing the following command:

cd C:\rclone

In order to test the connection you configured, execute the command below. Replace remote-config with the name of the connection you just configured.

.\rclone.exe lsd remote-config:

You should see the list of object storage containers associated with your credentials, for example:

-1 2023-01-18 12:53:14        -1 second-test-container
-1 2023-01-16 13:23:03        -1 test-container

To test the mounting of one of your containers, execute the command below without leaving the PowerShell. Replace remote-config with the name of your connection, test-container with the name of your container and E: with the drive letter under which you wish to mount it.

.\rclone.exe mount remote-config:test-container E: --vfs-cache-mode full --dir-cache-time 1m0s
Warning
By default, EODATA repository is mounted on disk Y: so be sure to use some other letter for your drive.

The option in this command –vfs-cache-mode full should make the mount support standard file system operations.

The option –dir-cache-time 1m0s will be explained in the next step.

You should now get the following output:

The service rclone has been started.

Go to This PC window. You should see the mounted container there:

Enter it and you should see its content there:

To stop the test mount, press CTRL+C in the PowerShell. You should get the following output:

The service rclone has been stopped.

The container should no longer be visible in This PC window.

If pressing CTRL+C does not stop the test mount, make sure that the PowerShell window is focused by left-clicking it. Press a letter on your keyboard, for example A, and try pressing CTRL+C again.

You can perform test mounts for all object storage containers you wish to access on your virtual machine.

Do not close PowerShell yet.

Step 3: Tweak the –dir-cache-time option

In Step 2, you performed a test mount of your object storage container using the following command:

.\rclone.exe mount remote-config:test-container E: --vfs-cache-mode full --dir-cache-time 1m0s

Tweaking the option –dir-cache-time is important especially if you intend to use your container on multiple physical and/or virtual machines. This includes using the container on your virtual machine and the Horizon dashboard. You might discover that the changes made to the bucket on another computer do not appear on your Windows VM. Using the Refresh option of the Windows File Explorer might not synchronize that change either.

That is because the Refresh option in this case does not pull the changes directly from the container, but from the cache. If the option –dir-cache-time is not specified during mounting, the cache is automatically synchronized every 5 minutes. Therefore, if you for example change a name of the folder on your other device, you will be able to pull that change after up to about 5 minutes.

Specifying this option overwrites this default value of 5 minutes. In this example, the automatic refresh of cache was set to 1 minute (1m0s). It is also possible to set this value to for example 1 second (0m1s). You can replace 1m0s in the command above with the value of your choice.

You can now perform a few test mounts as explained in Step 2 and find the –dir-cache-time value that suits you.

Step 4: Configuring automatic mounting of your container

To configure automatic mounting of your drive after logging in to Windows, return to PowerShell.

While still in C:\rclone folder, execute the command below.

.\nssm.exe install

You should get the following window:

Click the … button next to the Path: text field.

Choose the location of Rclone. If you followed this tutorial, this location is as follows:

C:\rclone\rclone.exe

In the Arguments text field enter the following code. Replace remote-config with the name of your connection, test-container with the name of your container, E: with the drive letter under which you wish to mount it and 1m0s with the value you chose in Step 3.

mount remote-config:test-container E: --vfs-cache-mode full --dir-cache-time 1m0s
Warning
By default, EODATA repository is mounted on disk Y: so be sure to use some other letter for your drive.

In the text field Service name: enter the name for your mounting service. It can be different than the name of your connection you set in Step 1 and the name of your S3 container. In this example, the name mounting-service will be used.

Navigate to the Log on. Select the option This account:. In the text field next to that option enter Administrator. Enter the password for your Administrator account in the Password: and Confirm: text fields.

Click Install service.

Repeat the process for each object storage container you wish to have automatically mounted.

Restart your VM and check whether the drives gets automatically mounted in the This PC window. If it is, the service works as intended.

You should now be able to work with your files.

If you find yourself unable to delete files or folders on the object storage, you can remove them from the Object Store -> Containers option in the Horizon dashboard:

IMAGE

Disabling the automatic mounting of a container

If you no longer wish to access a container on a particular virtual machine, you need to disable its mounting. Open PowerShell and execute the following command there to navigate to the C:\rclone directory:

cd C:\rclone

To check the status of your automatic mounting service, execute the command below. Replace mounting-service with the name of your automatic mounting service you set in Step 4.

.\nssm.exe status mounting-service

You should get the following output:

SERVICE_RUNNING

To stop the automatic mounting of your container, execute the command below. Replace mounting-service as previously.

.\nssm.exe stop mounting-service

Delete the service by executing the command below. Replace mounting-service as previously.

.\nssm.exe remove mounting-service confirm

You should now get the output similar to this:

Service "mounting-service" removed successfully!

Repeat the process for each container you no longer wish to be mounted.

Open the C:\Users\Administrator\.config\rclone.conf file using Notepad or other plain text editor and remove the lines responsible for mounting of object storage you no longer wish to be mounted.

Important
The instructions for stopping the automatic mounting of the EODATA repository can be found here: How to mount eodata on Windows virtual machine on Creodias hosting.
Important
On virtual machines with automatic mounting of the EODATA repository preconfigured, the mounting process is done using a script and not using a service configured in NSSM. However, if you configured automatic mounting of the EODATA repository yourself as described in Method 1 of the article How to mount eodata on Windows virtual machine on Creodias hosting, you configured it using NSSM and the service you created is called EODATA.

What To Do Next

Object storage containers on the Creodias WAW3-1 cloud can be mounted both on physical and virtual machines running Windows and Linux. To mount the object storage container on a different platform, please follow one of the articles below:

How to mount object storage container from Creodias as file system on local Windows computer

How to Mount Object Storage Container as a File System in Linux Using s3fs on Creodias

How to access private object storage using S3cmd or boto3 on Creodias


How to use GUI in VM with Linux?

Note

This article assumes that you have access to CloudFerro WAW3-1 infrastructure.

If your CREODIAS account has access only to CF2 infrastructure, please contact support to get access to WAW3-1.

In this article you will learn how to use GUI (graphical user interface) on a Linux virtual machine running on Creodias WAW3-1 cloud.

For this purpose, you will install and use X2Go on your local Linux computer.

This article covers the installation of two desktop environments: MATE and XFCE. Choose the one that suits you best.

What We Are Going To Cover

  • Installing X2Go client

  • Installing X2Go server and desktop environment (MATE or XFCE)

  • Connecting to your virtual machine using X2Go client

  • Basic troubleshooting

Prerequisites

No. 1 Account

You need a Creodias hosting account with access to the Horizon interface: https://horizon.cloudferro.com/auth/login/?next=/.

No. 2 Linux installed on your local computer

You need to have a local computer with Linux installed. This article was written for such computers running Ubuntu Desktop 22.04. If you are running a different Linux distribution, adjust the instructions from this article accordingly.

No. 3 Linux virtual machine

You need a Linux virtual machine running on Creodias WAW3-1 cloud. You need to able to access it via SSH. The following article explains how to create one such virtual machine:

How to create a Linux VM and access it from Linux command line on Creodias

This article was written for virtual machines using a default Ubuntu 20.04 image on WAW3-1 cloud. Adjust the instructions from this article accordingly if your virtual machine has a different Linux distribution.

Step 1: Install X2Go client

Open the terminal on your local Linux computer and update your packages by executing the following command:

sudo apt update && sudo apt upgrade

Now, install the x2goclient package:

sudo apt install x2goclient

Step 2: Install the desktop environment on your VM

Method 1: Installing MATE

Connect to your VM using SSH. Update your packages there:

sudo apt update && sudo apt upgrade

Now, install the MATE desktop environment and X2Go server:

sudo apt install x2goserver ubuntu-mate-desktop mate-applet-brisk-menu

You can add other packages to that command as needed.

During the installation you will be asked to choose the keyboard layout. Choose the one that suits you best using the arrow keys and Enter.

Once the installation is completed, reboot your VM by executing the following command:

sudo reboot

Method 2: Installing XFCE

Connect to your VM using SSH. Update your packages there:

sudo apt update && sudo apt upgrade

Now, install the XFCE desktop environment, the terminal emulator and X2Go server:

sudo apt install x2goserver xfce4 xfce4-terminal

You can add other packages to that command as needed.

During the installation you will be asked to choose the keyboard layout. Choose the one that suits you best using the arrow keys and Enter.

Once the installation is completed, reboot your VM by executing the following command:

sudo reboot

Step 3: Connect to your VM using X2Go

Open X2Go on your local Linux computer. If you haven’t configured any session yet, you should get the window used for creating one:

If you didn’t get such window, click the New session button on the X2Go toolbar.

Enter the name of your choice for your session in the Session name: text box. In this example, the name cloud-session will be used.

In the text box Host: enter the floating IP of your VM.

In the Login: button enter eouser. If you are running the GNOME desktop environment on your local computer and you

Click the folder icon next to the Use RSA/DSA key for sh connection: text field. A file selector should appear. Choose the SSH private file you use for connecting to your VM via SSH.

From the drop-down menu in the Session type section choose the desktop environment you installed, for example MATE or XFCE.

In the Input/Output tab choose the screen resolution that suits you best in the Display section. Here you can also choose whether you wish to share clipboard with the remote VM in the Clipboard mode section.

Click OK. The window should close.

The session you created should now be visible in the X2Go client:

Click the name of your session to connect to your VM. Wait up to a minute until your connection is established. You should now see your desktop environment.

If you chose MATE, it should look like this:

If you, however, chose XFCE, it should look like this:

Troubleshooting - Using the terminal emulator on XFCE

If the button Terminal Emulator on your taskbar does not launch your terminal, click the Applications menu in the upper left corner of the screen:

Choose Settings -> Preferred Applications:

You should get the following window:

Open the tab Utilities. The window should now look like this:

From the drop-down menu in the Terminal Emulator section choose Xfce Terminal.

Click Close.

The button should now launch the terminal emulator correctly.

Troubleshooting - Keyboard layout

If you discover that the system does not use the keyboard layout you chose during the installation of the desktop environment, you will need to set it manually. The process differs depending on the desktop environment you chose.

MATE

Click the Menu in the upper left corner of the screen:

From the Preferences section choose Keyboard:

You should get the following window:

Navigate to the Layouts tab:

Here, you can add or remove keyboard layouts depending on your needs.

XFCE

From the Applications menu in the upper left corner of the screen choose Settings -> Keyboard. You should get the following window:

Go to the Layout tab:

Unselect the Use system defaults check box. You can now add or remove the keyboard layouts depending on your needs.

 


How to access EODATA and Object Storage using s3cmd (Linux)?

You can install s3cmd using Python PIP or from your Linux repository.

Installation from your system repository on Debian/Ubuntu systems:

Check for updates:

$ sudo apt update

Install s3cmd:

$ sudo apt install s3cmd

Installation from the Python repository (on most Linux distributions with python and pip preinstalled):

Installing with PIP:

Check if you have PIP installed:

$ pip

The program 'pip' is currently not installed. To run 'pip' please ask your administrator to install the package 'python-pip'

If it is not installed (Ubuntu):

$ sudo apt install python-pip

$ pip --version

pip 8.1.1 from /usr/lib/python2.7/dist-packages (python 2.7)

$ sudo pip install s3cmd

If you see the following:

Traceback (most recent call last):
File "/usr/bin/pip", line 11, in <module>
sys.exit(main())
File "/usr/lib/python2.7/dist-packages/pip/init.py", line 215, in main
locale.setlocale(locale.LC_ALL, '')
File "/usr/lib/python2.7/locale.py", line 581, in setlocale
return _setlocale(category, locale)
locale.Error: unsupported locale setting

add the following line:

export LC_ALL=en_US.UTF-8

to the file:

~/.profile

Now you can check the .profile:

$ cat ~/.profile
export LC_ALL=en_US.UTF-8
$ source ~/.profile
$ s3cmd --version
s3cmd version 2.0.1

s3cmd configuration

Enter the following command:

$ s3cmd --configure

Refer to the user manual for detailed description of all options.

Access key and Secret key are your identifiers for Amazon S3. Insert following credentials to grant access to EODATA:

Access Key [access]:CLOUDFERRO
Secret Key [access]:PUBLIC
Default Region [RegionOne]: default
Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3.
S3 Endpoint [data.cloudferro.com:] data.cloudferro.com
Use "%(bucket)s.s3.amazonaws.com" to the target Amazon S3. "%(bucket)s" and "%(location)s" vars can be used
if the target S3 system supports dns based buckets.
DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]:  <ENTER>
Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password: <ENTER>
Path to GPG program [/usr/bin/gpg]: <ENTER>
When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP, and can only be proxied with Python 2.7 or newer
Use HTTPS protocol [No]: False
On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't connect to S3 directly
HTTP Proxy server name: <ENTER>

New settings:

       Access Key: CLOUDFERRO
       Secret Key: PUBLIC
       Default Region: default
       S3 Endpoint: data.cloudferro.com
       DNS-style bucket+hostname:port template for accessing a bucket: %(bucket)s.s3.amazonaws.com
       Encryption password:
       Path to GPG program: /usr/bin/gpg
       Use HTTPS protocol: False
       HTTP Proxy server name: _____
       HTTP Proxy server port: 0

     Test access with supplied credentials? [Y/n] <ENTER>
     Please wait, attempting to list all buckets...
     Success. Your access key and secret key worked fine :-)
     Now verifying that encryption works...
     Not configured. Never mind.

     Save settings? [y/N]  y <ENTER>
     Configuration saved to '/home/eouser/.s3cfg'

Now you can use s3cmd commands (additional information about s3cmd: http://s3tools.org/usage).

$ s3cmd ls

2017-12-11 15:30  s3://DIAS
2017-12-11 15:30  s3://EOCLOUD
2017-12-11 15:30  s3://EODATA

$ s3cmd ls s3://EODATA/

                       DIR   s3://EODATA/Envisat/
                       DIR   s3://EODATA/Landsat-5/
                       DIR   s3://EODATA/Landsat-7/
                       DIR   s3://EODATA/Landsat-8/
                       DIR   s3://EODATA/Sentinel-1/
                       DIR   s3://EODATA/Sentinel-2/
                       DIR   s3://EODATA/Sentinel-3/
                       DIR   s3://EODATA/Sentinel-5P/

In order to acquire access to Object Storage buckets via s3cmd, first you have to generate your own ec2 credentials with this tutorial How to Generate EC2 Credentials on Creodias.

After the creation of credentials please remove the file .s3cfg from your Home folder and then reconfigure s3cmd by entering:

s3cmd --configure

and the following values:

New settings:
Access Key: (your EC2 credentials)
Secret Key: (your EC2 credentials)
Default Region: none
S3 Endpoint: s3.waw3-1.cloudferro.com
DNS-style bucket+hostname:port template for accessing a bucket: s3.waw3-1.cloudferro.com
Encryption password: (your password)
Path to GPG program: /usr/bin/gpg
Use HTTPS protocol: True
HTTP Proxy server name:
HTTP Proxy server port: 0

After this operation, you should be allowed to list and access your Object Storage.


How to order new VM?

Login to https://portal.creodias.eu/clientarea.php

In the Client Area select Order/Buy option (highlighted in yellow hereunder).

Keep in mind, that resources such as instances or volumes are the paid ones and thus a sufficient amount of credits is required in order to buy them in a Pay Per Use billing. If you haven't bought any credits before, you can refer to this guide to do so.



Figure 1 - Creodias Client Page

 

Choose one product (in the example hereunder eo2.xlarge 4 vCores RAM 16GB lnetwork SSD storage 64GB) by selecting “Order Now” (highlighted in yellow). 

Please remember that the type of storage as well as the type of CPU you get with your VM depends on which machine flavor you choose. For example HMD flavors are equipped with NVMe local disks instead of network storage. For more info about flavors: Computing .



Figure 2 - Server selection page

 

„Configure” page lets you choose billing mode, choose whether you want to have your VM with EO data access and requires you to fill in information on instance’s name and network.

 


Figure 3 - Service configuration page

 

In the field hereunder you are ought to fill in the public key, corresponding to your private key, that you will be using to SSH to your VM. You can use our built-in SSH Key Generator by clicking on this link: https://portal.creodias.eu/ssh.php



Figure 4 - Public SSH key

 

After successful configuration of your VM, review page will appear. It summarizes the purchase.



Figure 5 - Creodias Checkout screen

 

In order to finalize the operation, select “complete order” option (highlighted hereunder)



Figure 6 - Creodias checkout page

 

Upon completion of the order, you will see order confirmation with a according order number as well as you will receive three e-mails.

 

Figure 7 - Customer Invoice



Figure 8 - Order confirmation



Figure 9 - Invoice payment confirmation

In case you ordered a VM with Fixed term billing, your environment will be initialized the moment you pay for it. To do so simply navigate to "My invoices " tab (https://portal.creodias.eu/clientarea.php?action=invoices) and click on the invoice you would like to pay.

 

You'll be presented with pro-forma invoice which you can pay for using credits from your Creodias account. Click "Apply credit" then your invoice status will turn as "Paid" and your environment will be initialized.

In order to manage you VM, you need to login to CloudFerro dashboard (https://cf2.cloudferro.com), using information from your registration process.

You will see OpenStack dashboard as depicted hereunder.

 

 
Figure 10 - OpenStack Dashboard

 

Choose Project/Compute/Instances

 

 
Figure 11 - Instances

 

You can see the Network Topology after choosing Network/Network Topology panel

 

Figure 12 - Network Topology

 

Choose Project/Compute/Instances, open drop-down menu and click “Associate Floating IP” option
 

Figure 13 - Associate Floating IP

 

Select IP Address (from the drop-down menu)

 

 
Figure 14 - Floating IP

 

Select Port to be associated and “Associate” it.

 

 
Figure 15 - Floating IP

 

You can see Floating IP addresses associated to your instances after choosing the Project/Computer/Instances

 

 
Figure 16 - Instances

 

Now you can SSH to your VM using the following linux command:

$ SSH -I your_private_key eouser@185.52.195.159

or using PuTTy for Windows


The CREODIAS would like to place cookies on your computer to help make this website better. By continuing to use the site you are agreeing to our use of cookies. If you do not agree with the terms and conditions, please do not use the website. Learn More... Accept