How to access private object storage using s3cmd or boto3?

Before attempting to perform operations concerning this FAQ, please familarize yourself with:

S3cmd

FAQ for accessing EO DATA:

How to access EO DATA and Object Storage using s3cmd (Linux)

S3cmd is installed on your virtual machine by default.

Use this command to create a configuration file (.s3cfg):

s3cmd --configure

Please follow those steps in order to accomplish configuration:

Enter new values or accept defaults in brackets with Enter.

Refer to user manual for detailed description of all options.


Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables.

Access Key: entry your access point

Secret Key: entry your secret key

Default Region [US]: RegionOne


Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3.

S3 Endpoint [s3.amazonaws.com]: s3.waw2-1.cloudferro.com


Use "%(bucket)s.s3.amazonaws.com" to the target Amazon S3. "%(bucket)s" and "%(location)s" vars can be used

if the target S3 system supports dns based buckets.

DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]: s3.waw2-1.cloudferro.com


Encryption password is used to protect your files from reading

by unauthorized persons while in transfer to S3

Encryption password:

Path to GPG program [/usr/bin/gpg]:


When using secure HTTPS protocol all communication with Amazon S3

servers is protected from 3rd party eavesdropping. This method is

slower than plain HTTP, and can only be proxied with Python 2.7 or newer

Use HTTPS protocol [Yes]:


On some networks all internet access must go through a HTTP proxy.

Try setting it here if you can't connect to S3 directly

HTTP Proxy server name:

Make sure about the values you have filled in a while ago.

New settings:

Access Key: insert your access key

Secret Key: insert your secret key

Default Region: RegionOne

S3 Endpoint: s3.waw2-1.cloudferro.com

DNS-style bucket+hostname:port template for accessing a bucket: s3.waw2-1.cloudferro.com

Encryption password:

Path to GPG program: /usr/bin/gpg

Use HTTPS protocol: True

HTTP Proxy server name:

HTTP Proxy server port: 0


Test access with supplied credentials? [Y/n] y

Please wait, attempting to list all buckets...

Success. Your access key and secret key worked fine :-)


Now verifying that encryption works...

Not configured. Never mind.


Save settings? [y/N] y
Configuration saved to '/home/eouser/.s3cfg'

In order to list available buckets type in command presented below:

eouser@vm1:~$ s3cmd ls
2019-01-03 16:20  s3://testjohn

To examine a particular bucket add its path:

eouser@vm1:~$ s3cmd ls s3://testjohn

2019-01-18 09:22 81712 s3://testjohn/console.png

2019-01-07 09:57 2589 s3://testjohn/tes1.txt

Boto3

FAQ for accessing EO DATA:

How to access/list EO DATA using boto3?

How to download EO DATA file using boto3?

Boto3 module installation:

We strongly recommend using virtualenv for isolating python packages. Configuration tutorial is presented below:

How to install Python virtualenv/virtualenvwrapper?

If virtualenv is activated:

$ pip install boto3 #for Python2
$ pip3 install boto3 #for Python3

Or if we install package with global entry point:

#Pipsi installation
$ curl https://raw.githubusercontent.com/mitsuhiko/pipsi/master/get-pipsi.py | python
$ pipsi install boto3

Simple script for accessing your private bucket:

import boto3

def boto3connection(access_key,secret_key,bucketname):
    host='https://s3.waw2-1.cloudferro.com'
    s3=boto3.resource('s3',aws_access_key_id=access_key,
aws_secret_access_key=secret_key, endpoint_url=host,)

    bucket=s3.Bucket(bucketname)

    for obj in bucket.objects.filter():
        print('{0}:{1}'.format(bucket.name, obj.key))

#For Python2
x = raw_input('Enter your access key:')
y = raw_input('Enter your secret key:')
z = raw_input('Enter your bucket name:')

#For Python3
x = input('Enter your access key:')
y = input('Enter your secret key:')
z = input('Enter your bucket name:')

boto3connection(x,y,z)

Save your file with .py extension and run with the "python [filename.py]" command in your terminal.

For Python3 use "python3 [filename.py]".

TAGS