We are regularly expanding our portfolio of available types for virtual machines. This time we introduce new configurations with ultra-fast, local, NVMe storage - HMD.

This family of VM offers next level disc performance for VMs with was previously available only in DS configurations (Dedicated server virtual machines).

The new configurations with local storage offers more than 10x better IOPS performance and 10x lower write and read times comparing to the previously available virtual machines. We decided for the new configurations to keep the CPU to RAM ratio aligned with HM line. 

HMDs are designed for such applications as:

• data processing requiring fast local cache,
• control nodes for Kubernetes clusters with fast storage for etcd,
• very fast data entry, handling of events from IoT devices,
• very fast saving of calculation results,
• hosting nonrelational databases.

While the input and output files are valuable assets and are typically stored as S3 objects very fast location designed for saving of logs, intermediate files storage, partial data files or even small chunks of results on hi performance storage can significantly improve processing time and optimize the use of infrastructure.

The new storage is the first ephemeral storage in our offer, its use is subject to several limitations such as lack of shelving or being limited to only one disc per VM. 

You can order servers in the Pay-per-use model through the openstack API or Horaizon or in long contracts through our e-commerce system. Cloud resources prices, in particular storage prices, are available in our price-list.

Learn more about different types of storage and its characteristics. Read also about the storage performance.

If you have problems with access to eodata try the following:

install arping:

in CentOS:

sudo yum install arping

in Ubuntu:

sudo apt install arping

check the name of the interface connected to eodata network:

ifconfig

based on the response, find the number of  the interface of 10.111.x.x (eth<number> or ens<number>)

after that invoke the following commands:

in CentOS:

sudo arping -U -c 2 -I eth<number> $(ip -4 a show dev eth<number> | sed -n 's/.*inet \([0-9\.]\+\).*/\1/p')

in Ubuntu:

sudo arping -U -c 2 -I ens<number> $(ip -4 a show dev ens<number> | sed -n 's/.*inet \([0-9\.]\+\).*/\1/p')

Next ping data.cloudferro.com again. If you receive answers, remount the resource.

In the instances created after March 14, 2020, /eodata is automatically mounted after adding the eodata web interface. The /eodata mount runs as a service and can be restarted by executing the command:

sudo systemctl restart eodata.mount

and status can be checked by executing:

sudo systemctl status eodata.mount

In the instances created before March 14, 2020, /eodata is mounted using the /etc/fstab file and can be remounted by executing commands below:

sudo umount -lf /eodata
sudo mount /eodata

in Windows:

in command line run as administrator:

route add 10.97.0.0/16 10.11.0.1

 

and than run "mount_eodata" script from desktop.

1. We assume that you have already registered your account in CREODIAS and you are logged in to Finder (https://finder.creodias.eu/).

2. In Finder, please choose MODIS Download in the "order products using" pane, as well as MODIS/TerraAqua as a collection. You will be able to limit products to the particular product type. You can also specify the area of interest and timelapse as in a regular Finder query (please see https://creodias.eu/-/how-to-use-creodias-finder-)

3. After specifying the query, click the "search" button, and the list of products will appear. You can choose the product by clicking on it and selecting the "add" option, or by selecting multiple products and clicking the "add all to cart" option. After finishing the selection, please click the "cart" button in the top right corner.

4. You will be asked for your s3 credentials (see how: https://creodias.eu/-/how-to-generate-ec2-credentials-), your Object Storage bucket and prefix (destination directory) name (please see: https://creodias.eu/-/a-9-36 and https://creodias.eu/-/how-to-access-private-object-storage-using-s3cmd-or-boto3-). After filling these sections, please specify the order name and priority and hit the "order" button.

5. You can check the status of your request by selecting the "orders" menu, next to the cart button. If the ordering status is done, the product should be downloaded to your object storage container.

S3 Bucket Policy

Ceph - the Software Defined Storage used in CloudFerro clouds, provides object storage compatibility with a subset of Amazon S3 API. Bucket policy allows for a selective access sharing to object storage buckets between users of different projects in the same cloud.

Naming convention used in this document

• Bucket Owner - OpenStack tenant who created an object storage bucket in their project, intending to share to their bucket or a subset of objects in the bucket to another tenant in the same cloud.
• Bucket User - OpenStack tenant who wants to gain access to a Bucket Owner's object storage bucket.
• Bucket Owner's Project - a project in which a shared bucket is created.
• Bucket User's Project - a project which gets access to Bucket Owner's object storage bucket.
• Tenant Admin - a tenant's administrator user who can create OpenStack projects and manage users and roles within their domain.
• In code examples, values typed in all-capital letters, such as BUCKET_OWNER_PROJECT_ID, are placeholders which should be replaced with actual values matching your use-case. 

Limitations

It is possible to grant access at the project level only, not at the user level. In order to grant access to an individual user, a Bucket User's Tenant Admin must create a separate project within their domain, which only selected users will be granted access to.

Ceph S3 implementation supports the following S3 actions by setting bucket policy.

Ceph S3 implementation does not support user, role or group policies.

Declaring bucket policy

Policy JSON file's sections

Bucket policy is declared using a JSON file. An example policy JSON template:

{
 "Version": "2012-10-17",
 "Id": "POLICY_NAME",
 "Statement": [
   {
     "Sid": "STATEMENT_NAME",
     "Effect": "EFFECT",
     "Principal": {
       "AWS": "arn:aws:iam::PROJECT_ID:root"
     },
     "Action": [
       "ACTION_1",
       "ACTION_2"
     ],
     "Resource": [
       "arn:aws:s3:::KEY_SPECIFICATION"
     ]
   }
 ]
}

Bucket policy description follows:

Setting a policy on a bucket

The policy may be set on a bucket using s3cmd setpolicy POLICY_JSON_FILE s3://MY_SHARED_BUCKET command. See S3 Tools s3cmd usage for complete documentation.

After installing s3cmd, initiate its configuration, by issuing: s3cmd --configure -c s3cmd-config-file.

A sample s3cmd config file in CloudFerro CF2 cloud:

[default]
access_key = MY_ACCESS_KEY
secret_key = MY_SECRET_KEY
bucket_location = RegionOne
host_base = s3.waw2-1.cloudferro.com
host_bucket = s3.waw2-1.cloudferro.com
use_https = True
verbosity = WARNING
signature_v2 = False

The access key and the secret key may be generated using openstack-cli:

openstack ec2 credentials create
+------------+-------------------------------------+
| Field      | Value                               |
+------------+-------------------------------------+
| access     | [access key]                        |
| links      | [link]                              |
| project_id | db39778a89b242f0a8ba818eaf4f3329    |
| secret     | [secret key]                        |
| trust_id   | None                                |
| user_id    | 121fa8dadf084e2fba46b00850aeb7aa    |
+------------+-------------------------------------+

Assuming that Bucket Owner's s3cmd config file name is 'owner-project-s3cfg', a simple example of setting policy follows:

s3cmd -c owner-project-s3cfg setpolicy sample-policy.json s3://mysharedbucket

To check policy on a bucket, use the following command:

s3cmd -c owner-project-s3cfg info s3://mysharedbucket

Deleting the policy from a bucket

The policy may be deleted a bucket using s3cmd setpolicy POLICY_JSON_FILE s3://MY_SHARED_BUCKET command. See S3 Tools s3cmd usage for complete documentation.

A simple example of setting policy follows:

s3cmd -c owner-project-s3cfg delpolicy s3://mysharedbucket

Sample scenarios

Grant another project access to read and write to the bucket

A Bucket Owner wants to grant a bucket a read/write access to a Bucket User.

{
  "Version": "2012-10-17",
  "Id": "read-write",
  "Statement": [
    {
      "Sid": "project-read-write",
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::BUCKET_OWNER_PROJECT_ID:root",
          "arn:aws:iam::BUCKET_USER_PROJECT_ID:root"
        ]
      },
      "Action": [
        "s3:ListBucket",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:GetObject"
      ],
      "Resource": [
        "arn:aws:s3:::*"
      ]
    }
  ]
}

To apply the policy, the Bucket Owner should issue:

s3cmd -c owner-project-s3cfg setpolicy read-write-policy.json s3://mysharedbucket

The Bucket Owner should send to the Bucket User information about Bucket Owner's project id and the name of the bucket.

After Bucket User prepared their s3cmd config file called 'user-project-s3cfg', to access the bucket, for example to list the bucket, the Bucket User should issue:

s3cmd -c user-project-s3cfg ls s3://BUCKET_OWNER_PROJECT_ID:mysharedbucket

Grant any user read access to the bucket

A bucket Owner wants to grant a bucket read access to anyone.

{
  "Version": "2012-10-17",
  "Id": "policy-read-any",
  "Statement": [
    {
      "Sid": "read-any",
      "Effect": "Allow",
      "Principal": {
        "AWS": [
		   "*"
		]
      },
      "Action": [
        "s3:ListBucket",
        "s3:GetObject"
      ],
      "Resource": [
        "arn:aws:s3:::*"
      ]
    }
  ]
}

To apply the policy, the Bucket Owner should issue:

s3cmd -c owner-project-s3cfg setpolicy read-any-policy.json s3://mysharedbucket

The Bucket Owner should publish the Bucket Owner's project id and the name of the bucket.

Users from other projects can access to bucket's contents, for example retrieve pictures/mypic.png from the Bucket Owner's bucket:

s3cmd -c user-project-s3cfg get s3://BUCKET_OWNER_PROJECT_ID:mysharedbucket/pictures/mypic.png

Grant one user write access and another user read access to a subfolder of a bucket

A Bucket Owner wants to share a folder in a bucket with read/write permissions to First Bucket User and read permissions to Second Bucket User.

{
    "Version": "2012-10-17",
    "Id": "complex-policy",
    "Statement": [
        {
            "Sid": "project-write",
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::BUCKET_OWNER_PROJECT_ID:root",
                    "arn:aws:iam::FIRST_BUCKET_USER_PROJECT_ID:root"
                ]
            },
            "Action": [
                "s3:ListBucket",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::mysharedbucket/mysharedfolder/*"
            ]
        },
        {
            "Sid": "project-read",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::SECOND_BUCKET_USER_PROJECT_ID:root"
            },
            "Action": [
                "s3:ListBucket",
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::mysharedbucket/mysharedfolder/*"
            ]
        }
    ]
}

To apply the policy, the Bucket Owner should issue:

s3cmd -c owner-project-s3cfg setpolicy complex-policy.json s3://mysharedbucket

The Bucket Owner should send to the First and Second Bucket Users information about Bucket Owner's project id and the name of the bucket.

To access the bucket, for example to write productlist.db to the bucket, the First Bucket User should issue:

s3cmd -c first-user-project-s3cfg put productlist.db s3://BUCKET_OWNER_PROJECT_ID:mysharedbucket/mysharedfolder/

The Second Bucket User can read produclist.db from the bucket:

s3cmd -c second-user-project-s3cfg get s3://BUCKET_OWNER_PROJECT_ID:mysharedbucket/mysharedfolder/productlist.db

Access to shared buckets with boto3

The following python script shows how Bucket User can list objects in Bucket Owner's bucket mybucket using boto3 library.

#!/usr/bin/python3


import boto3
from botocore.session import Session
from botocore.handlers import validate_bucket_name


ACCESS_KEY = "Bucket User's Access Key"
SECRET_KEY = "Bucket User's Secret Key"
ENDPOINT = "https://s3.waw2-1.cloudferro.com"
BUCKET_OWNER_PROJECT_ID = "Bucket Owner's Project ID"
BUCKET = "mybucket"

bucket_location = BUCKET_OWNER_PROJECT_ID + ":" + BUCKET

# We need to skip bucket name validation due to
# multitenancy Ceph bucket naming: tenantId:bucket.
# Otherwise we will receive an exception:
# "Parameter validation failed: Invalid bucket name".
botocore_session = Session()
botocore_session.unregister('before-parameter-build.s3', validate_bucket_name)
boto3.setup_default_session(botocore_session = botocore_session)

s3 = boto3.client(
    's3',
    aws_access_key_id=ACCESS_KEY,
    aws_secret_access_key=SECRET_KEY,
    endpoint_url=ENDPOINT,
)

response = s3.list_objects(Bucket=bucket_location)

print(response)

Please note that since AWS S3 standard does not support colons in the bucket names and here we address bucket names as tenantId:bucket, we have to workaround botocore validator, which checks if the bucket name matches regular expression: ^[a-zA-Z0-9.\-_]{1,255}$. Hence the extra line in the code:

botocore_session.unregister('before-parameter-build.s3', validate_bucket_name)
boto3.setup_default_session(botocore_session = botocore_session)

Without the code, the bucket name validator raises an exception:

botocore.exceptions.ParamValidationError: Parameter validation failed:
Invalid bucket name "TENANT_ID:BUCKET_NAME": Bucket name must match the regex "^[a-zA-Z0-9.\-_]{1,255}$"
or be an ARN matching the regex "^arn:(aws).*:s3:[a-z\-0-9]+:[0-9]{12}:accesspoint[/:][a-zA-Z0-9\-]{1,63}$
|^arn:(aws).*:s3-outposts:[a-z\-0-9]+:[0-9]{12}:outpost[/:][a-zA-Z0-9\-]{1,63}[/:]accesspoint[/:][a-zA-Z0-9\-]{1,63}$"

 References

• Ceph: Bucket Policy documentation
• RedHat: Object Gateway S3 Application Programming Interface (API)
• Amazon: Using Bucket Policies and User Policies
• S3 Tools s3cmd usage