How to mount EODATA as a filesystem using Goofys in Linux on CREODIAS

In this article, you will mount the EODATA repository using goofys on your Linux virtual machine.

This article was written for virtual machines running on CREODIAS CF2 and WAW3-1 cloud. WAW3-2 and FRA1-2 clouds are not covered by this article.

The EODATA repository contains dozens of petabytes of Earth observation data from missions such as Sentinel-2A or Sentinel-5P.

Goofys is a piece of software used for mounting S3 containers. You will use it for mounting the bucket containing the EODATA repository on your Linux VM.

On CREODIAS cloud the virtual machines come configured mounting of the EODATA repository using s3fs. The following article contains more information: How to mount eodata using s3fs in Linux on CREODIAS. This article, on the other hand, is for users who want to use goofys.

Prerequisites

No. 1 Account

You need a CREODIAS hosting account with access to the Horizon interface: https://horizon.cloudferro.com.

No 2. Linux VM

You will need a Linux VM on the CREODIAS cloud with access to the EODATA network. You can create it by following one of the articles:

This article was created for Ubuntu 20.04. If you are using a different Linux distribution, adjust the commands accordingly.

No 3. Basic knowledge of Linux command line

Basic knowledge of Linux command line is required.

What We Are Going To Cover

  • Installing goofys

  • Providing credentials to configure goofys

  • Mounting the EODATA repository

  • Configuring automatic mounting of EODATA

  • Stopping the automatic mounting of EODATA repository using goofys

Step 1: Install goofys

Login to the eouser account of your virtual machine using SSH.

Install wget to be able to download the goofys binary directly to your virtual machine:

sudo apt update && sudo apt upgrade && sudo apt install wget

Now, navigate to your home directory:

cd

Use wget to download the goofys binary:

wget https://github.com/kahing/goofys/releases/latest/download/goofys

Make the goofys binary executable:

chmod +x goofys

To update goofys in the future:

  • make sure that the data in the EODATA repository is not actively used by any software on your virtual machine,

  • remove the goofys binary from your home directory,

  • execute the above commands (those starting with wget and chmod) from your home directory again and

  • reboot your virtual machine.

Step 2: Provide credentials to configure goofys

Make a folder to store your credentials:

mkdir ~/.aws/

Create file ~/.aws/credentials using your favorite text editor (for example nano or vim). Add the following contents to it:

[s3]
aws_access_key_id=CLOUDFERRO
aws_secret_access_key=PUBLIC

Save the file and exit the text editor.

Step 3: Mount EODATA

Create the directory which will be the mount point for EODATA, for example:

mkdir ~/eo

Mount EODATA using the following command:

./goofys --region RegionOne --profile s3 --endpoint http://data.cloudferro.com DIAS ~/eo

In this command,

  • ./goofys is the location of the goofys binary you downloaded

  • ~/eo is the location of the folder in which you want to mount the EODATA filesystem

  • DIAS is the name of the bucket which contains the EODATA resources.

You can replace the first two values listed above as needed.

In order to test whether the mount was successful, navigate to the directory in which you mounted the EODATA repository, for example:

cd ~/eo

Use the ls command to list its content. You should see the output similar to this:

../_images/mount-eodata-goofys-01_creodias.png

The EODATA repository has now been mounted using goofys.

After each reboot, you will need to execute the mounting command above again.

Step 4 Mounting EODATA on startup

Here is how to automatically mount the EODATA repository using goofys at system startup.

If you still have the EODATA repository mounted using the method described in Step 3, reboot your virtual machine.

Create directory in /root folder in which you will store the credentials:

sudo mkdir /root/.aws

Copy the credentials you created in your local directory to the .aws directory in the /root folder:

sudo cp ~/.aws/credentials /root/.aws/

Now, add the line below to the file /etc/fstab. In order to do that, you need to start your text editor with elevated privileges (for example using sudo).

/home/eouser/goofys#DIAS /home/eouser/eo fuse _netdev,allow_other,--dir-mode=0777,--file-mode=0666,--region=RegionOne,--profile=s3,--endpoint=http://data.cloudferro.com 0 0

In the above command,

  • /home/eouser/goofys is the goofys binary you downloaded and

  • /home/eouser/eo is the location in which you wish to mount EODATA.

Once you have added that line to your /etc/fstab file, reboot the VM. After the system has restarted, check whether the EODATA repository is mounted in the directory specified by you.

Stopping the automatic mounting of EODATA repository using goofys

If you want to stop automatic mounting of the EODATA repository, remove the line you added in Step 3 from /etc/fstab file. You can also comment it out by adding # character in front of that line.a

After that, reboot the VM. Optionally, you can also remove the goofys binary and the credentials if you no longer want to use goofys.

What To Do Next

You can also try other methods of accessing EODATA, such as:

How to mount eodata using s3fs in Linux on CREODIAS

How to download EODATA files using s3cmd on CREODIAS

If you want to access EODATA on a Windows VM, you can do it by following this article:

How to mount EODATA on Windows virtual machine on CREODIAS hosting

If you find that the amount of disk space on your VM is insufficient for your EODATA processing, you can create and attach a volume to it. You can do so by following one of these articles: