Recommended usage of access interfaces to EO DATA
There are four data access interfaces used for accessing and downloading products from EO Data repository:
- S3FS - emulated file based access over S3 interface
- NFS - file based access
- S3 - object storage based interface - the recommended data access interface in CREODIAS
- Download - zipped products from https://finder.creodias.eu
Apart from the way of data access, these interfaces differ in data transfer speed and reliability.
While S3FS is a stable file based access to EO Data repository, it is not so fast. You may expect transfers of about 30 MB/s. Please note that S3FS does not work in Windows virtual instances since there is no S3FS client for Windows. There are some commercial alternatives such as https://tntdrive.com/ or https://www.cloudberrylab.com/drive.aspx. These alternatives are neither provided in CREODIAS Windows images nor covered by CREODIAS support.
With S3FS you may expect throughput of c.a. 30 MB/s.
S3FS is accessible only from virtual instances in CloudFerro clouds.
S3FS is the default file-based access in CREODIAS Linux instances.
NFS is much faster than S3FS file based access to EO Data, but not as stable, therefore sometimes one may experience "Stale file handle" or "Permission denied" errors. In such case it is recommended to remount the share.
With NFS you may expect at least 80-120 MB/s and more. A failure rate of ~0,1% in access to data should be handled on the application level. In case f a failure it is reFor instance remount the share or retry access to data instead of failing the application.
NFS is accessible only from virtual instances in CloudFerro clouds.
NFS access is available by default for Windows instances.
If you have a possibility to access EO Data directly with S3 object storage interface from your application, you might use boto3 library in Python scripts (How to download EO Data file using boto3) or other S3 libraries depending on your programming language. It is the fastest and most efficient way to access EO Data.
S3 provides throughput up to 200 MB/s for applications over API access or via s3cmd command line tool. (How to access EO DATA and Object Storage using s3cmd (Linux))
S3 interface is accessible only from virtual instances in CloudFerro clouds.
S3 is the recommended data access interface for EO Data in CREODIAS.
Users who don't have resources in CloudFerro clouds, may use CREODIAS Finder (https://finder.creodias.eu) to download complete EO products from EO Data repository in form of a single ZIP file.
The expected throughput of the download interface is about 25 MB/s.
Products downloading is available for every registered CREODIAS user.