Forum

Message Boards Message Boards

Back

Sentinel 3 Downloading Issues

Sentinel 3 Downloading Issues
Answer
23 July 2021 9:13
Hi,

I've searched far and wide for a way to access a rather large time-series of Sentinel 3 SLSTR LST data. I've tried Sentinelsat in Python, but unfortunately, it has not been very succesful, as copernicus has most of the Sentinel 3 data stored offline, and I can only recover 20 scenes at a time with my login credentials. Given I'm looking for a 2016-2021 time-series covering the entire Country of Canada, 20 scenes at a time is not a realistic approach for me to get the data. 

I'm hoping I can work around that issue with Creodias?
I was able to query my data, but no processor is available, so I cannot click the 'add all to cart' option. 

I'm quite new to using APIs to download data, and was hoping someone here might have a sample python script for searching and downloading data from Creodias. 

Thanks in advance for the help!
0 (0 Votes)

RE: Sentinel 3 Downloading Issues
Answer
26 July 2021 7:45 as a reply to Ramon Melser.
Hi Ramon,

Since you want to access a lot of data, maybe processing them using a virtual machine directly on the Creodias platform would be more efficient.

If you want to fetch the files, the script below should allow you to download all the data that match a Creodias finder search request:

1. Go to Creodias finder (https://finder.creodias.eu), select search criteria and submit search request
2. Check that the results only contain what you want to download (the script will download all matching results so you really should avoid false positives)
3. Copy the search URL (the field with a "Rest query" placeholder, above the "Polygon Selection" and "Upload Polygon" buttons)
4. Execute the script with the URL as argument (you probably need to wrap the URL in single quotes to avoid issues)
5. The script will ask for your Creodias credentials (required for download)

You should probably proceed with a small time frame first to check that it works correctly.

Note that the script relies on Python, awk, curl and wget so make sure you have these three tools available on your system.
12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273

#!/bin/bash

# Script to perform search requests on Creodias finder and automatically
# download the results.
#
# @author: <sylvain.herledan@oceandatalab.com>
# @date: 2021-05-21
#
#
# Usage:
#
#    ./creodias_search_and_download.sh 'SEARCH_URL'
#
# where:
#  - SEARCH_URL is the URL containing all the search parameters
#
# The command will ask for a Creodias username and password (required for
# downloading data from Creodias).
#
# Note that the max number of results will be automatically increased from 10
# (default on Creodias finder) to 500 (max supported value) so that it is not
# required to edit this field manually when copying the URL from Creodias
# finder web interface.

set -o nounset
set -o errexit
set -o pipefail

readonly PROGDIR="$(readlink -f "$(dirname "${0}")")";
readonly PROGNAME="$(basename "${0}")";
readonly ARGS="${@}";

main()
{
  local username;
  local password;
  local search_url;
  local keycloak_token;

  search_url="${1}"; shift;
 
  read -p "Creodias username: " -r username;
  read -p "Creodias password: " -r -s password;
  echo

  curl -XGET --silent "$(echo "${search_url}" \
                         | sed 's/maxRecords=10/maxRecords=500/')" \
   | python -mjson.tool \
   | grep '"url": ' \
   | sed -e 's/^.*"url": "//g' -e 's/",.*$//g' \
   | while read -r download_url;
     do
       # Get a new token for each download (maybe seem unefficient but tokens
       # expire after some time, so using a single token for all downloads
       # will work for the first few downloads and then fail...
       keycloak_token="$(curl -d 'client_id=CLOUDFERRO_PUBLIC' \
                              -d "username=${username}" \
                              -d "password=${password}" \
                              -d 'grant_type=password' \
                              --silent \
                              'https://auth.creodias.eu/auth/realms/DIAS/protocol/openid-connect/token' \
                         | python -m json.tool \
                         | grep "access_token" \
                         | awk -F\" '{print $4}')";

       echo "Downloading ${download_url} ..."
       wget --quiet --content-disposition "${download_url}?token=${keycloak_token}"
    done
}

main ${ARGS}
exit 0
0 (0 Votes)

RE: Sentinel 3 Downloading Issues
Answer
26 July 2021 5:01 as a reply to Sylvain Herlédan.
That certainly looks very promising. Thank you for taking the time to write that up!
This may be a basic question, but how would I go about using a virtual machine from Creodias? Is that a paid service?
0 (0 Votes)