Forum

Message Boards Message Boards

Back

Automatic Download (using python)?

Toggle
Automatic Download (using python)?
Answer
27 May 2020 2:04
Hey Guys,
I am trying to download a lot of data that I can get using the finder-API (https://finder.creodias.eu/resto/api/collections/Sentinel1/search.json?maxRecords=2000&startDate=2018-09-01T00:00:00Z&completionDate=2019-04-01T23:59:59Z&productType=GRD&sensorMode=IW&sortParam=startDate&sortOrder=descending&status=all&geometry=POLYGON((25.471802+58.031372000000005,25.471802+58.81658600000003,26.872559000000003+58.81658600000003,26.872559000000003+58.031372000000005,25.471802+58.031372000000005))&dataset=ESA-DATASET).

I can download all the scenes manually by clicking them in the finder and save them, but this would take a lot of time and feel dumb ;) . Is there a way I can auto-download the images, if possible maybe even using python? (Comparable to sentinelsat for the copernicus-hub?)

Today is my first contact with CREODIAS, thus I might just need a hint in the right direction because I didn't find the right information by now.

EDIT: I managed to download single files using requests.urllib.get() but unfortunately the token is changing thus I need to find a way to create one on a Windows-Machine

Thanks,
Felix
0 (0 Votes)

RE: Automatic Download (using python)?
Answer
3 June 2020 2:28 as a reply to Felix Keßler.
Hello Felix,

We've made a Python script for you, just update finder_api_url, username and password variables in below script:

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657
import requests
import json
# from pprint import pprint
from tqdm import tqdm
import re
import sys

def get_keycloak_token(username, password):
    h = {
    'Content-Type': 'application/x-www-form-urlencoded'
    }
    d = {
    'client_id': 'CLOUDFERRO_PUBLIC',
    'password': password,
    'username': username,
    'grant_type': 'password'
    }
    resp = requests.post('https://auth.creodias.eu/auth/realms/dias/protocol/openid-connect/token', data=d, headers=h)
    # print(resp.status_code)
    try:
        token = json.loads(resp.content.decode('utf-8'))['access_token']
    except KeyError:
        print("Can't obtain a token (check username/password), exiting.")
        sys.exit()
    # print(token)
    return token

finder_api_url = '''https://finder.creodias.eu/resto/api/collections/Sentinel1/search.json?maxRecords=5&startDate=2018-09-01T00:00:00Z&completionDate=2019-04-01T23:59:59Z&productType=GRD&sensorMode=IW&sortParam=startDate&sortOrder=descending&status=all&geometry=POLYGON((25.471802+58.031372000000005,25.471802+58.81658600000003,26.872559000000003+58.81658600000003,26.872559000000003+58.031372000000005,25.471802+58.031372000000005))&dataset=ESA-DATASET'''
username = 'creodias_username_here'
password = 'creodias_password_here'

response = requests.get(finder_api_url)
for feature in json.loads(response.content.decode('utf-8'))['features']:
    token = get_keycloak_token(username, password)
    download_url = feature['properties']['services']['download']['url']
    download_url = download_url + '?token=' + token
    total_size = feature['properties']['services']['download']['size']
    title = feature['properties']['title']
    filename = title + '.zip'
    r = requests.get(download_url, stream=True)
    if "Content-Disposition" in r.headers.keys():
        filename = re.findall("filename=(.+)", r.headers["Content-Disposition"])[0]
    # Total size in bytes.
    total_size = int(r.headers.get('content-length', 0))
    if total_size <= 100:
        print(r.text)
        sys.exit("Please try again in few moments.")
    block_size = 1024 #1 Kibibyte
    print('downloading:', filename)
    t=tqdm(total=total_size, unit='iB', unit_scale=True)
    with open(filename, 'wb') as f:
        for data in r.iter_content(block_size):
            t.update(len(data))
            f.write(data)
    t.close()
    if total_size != 0 and t.n != total_size:
        print("ERROR, something went wrong")

Keep in mind that the maximum products you can find, thus download with this script, at once is 2000. To overcome this add '&page=<page_number>' to the finder_api_url, and change <page_number> iterativly after each use of the script.


Regards,
Marcin
0 (0 Votes)

RE: Automatic Download (using python)?
Answer
4 June 2020 5:18 as a reply to Marcin Szreder.
Hello Marcin,
Thanks a lot for this script. Right now it is not working, I am not registered at Creodias itself but I am able to access the data through Authentification via Code-DE, thus my Username/Password-Combination is not valid for Creodias.
Is it possible to update the code for this to work as well?

Thanks a lot for your work!
Felix
0 (0 Votes)

RE: Automatic Download (using python)?
Answer
15 June 2020 7:15 as a reply to Felix Keßler.
Hey Marcin,
I was wondering if there is an Update or a workaround you could provide for my problem?
0 (0 Votes)

RE: Automatic Download (using python)?
Answer
15 June 2020 8:17 as a reply to Felix Keßler.
Hello

Sorry for late reply. Below is a modified script to use with CODE-DE platform.


123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

import requests
import json
# from pprint import pprint
from tqdm import tqdm
import re
import sys


def get_keycloak_token(username, password):
    h = {
        'Content-Type': 'application/x-www-form-urlencoded'
    }
    d = {
        'client_id': 'FINDER',
        'password': "*****",
        'username': "******",
        'grant_type': 'password'
    }
    resp = requests.post(
        'https://auth.cloud.code-de.org/auth/realms/code-de/protocol/openid-connect/token', data=d, headers=h)
    # print(resp.status_code)
    try:
        token = json.loads(resp.content.decode('utf-8'))['access_token']
    except KeyError:
        print("Can't obtain a token (check username/password), exiting.")
        sys.exit()
    # print(token)
    return token


finder_api_url = '''https://finder.code-de.org/resto/api/collections/Sentinel2/search.json?maxRecords=10&startDate=2020-05-28T00:00:00Z&completionDate=2020-05-29T23:59:59Z&location=all&sortParam=startDate&sortOrder=descending&status=all&geometry=POLYGON((12.209161052771645+51.41610114539586,12.207943043941432+51.242201496186965,12.549586691921304+51.243726461021595,12.548368729553374+51.41496171793736,12.209161052771645+51.41610114539586))&dataset=ESA-DATASET'''
username = 'creodias_username_here'
password = 'creodias_password_here'
response = requests.get(finder_api_url)
for feature in json.loads(response.content.decode('utf-8'))['features']:
    token = get_keycloak_token(username, password)
    download_url = feature['properties']['services']['download']['url']
    download_url = download_url + '?token=' + token
    total_size = feature['properties']['services']['download']['size']
    title = feature['properties']['title']
    filename = title + '.zip'
    r = requests.get(download_url, stream=True)
    if "Content-Disposition" in r.headers.keys():
        filename = re.findall(
            "filename=(.+)", r.headers["Content-Disposition"])[0]
    # Total size in bytes.
    total_size = int(r.headers.get('content-length', 0))
    if total_size <= 30:
        print("Please try again in few moments.")
        sys.exit()
    block_size = 1024  # 1 Kibibyte
    print('downloading:', filename)
    t = tqdm(total=total_size, unit='iB', unit_scale=True)
    with open(filename, 'wb') as f:
        for data in r.iter_content(block_size):
            t.update(len(data))
            f.write(data)
    t.close()
    if total_size != 0 and t.n != total_size:
0 (0 Votes)

RE: Automatic Download (using python)?
Answer
17 June 2020 8:45 as a reply to Marcin Szreder.
Hey Marcin,
I think I found the reason why it is not working, I think you misunderstood what I wanted to achieve (hopefully it is actually possible):
I am registered at CodeDE and can use this to "log in" to the CreodiasFinder and download data. CodeDE has some data stored, but unfortunately only for Germany. Since I want to use data outside of Germany I wanted to download the data from Creodias.
If I go to https://finder.creodias.eu/ and search for data I can log in using the "Federated Identity Provider"-Tab and there CodeDE. If I am logged in I can download the data manually.
The code you provided was for automation of download if I have a CodeDE-API query. But I want to download from Creodias using my CodeDE-login data.

I hope this is more precise now and you get what I want. I think every time your code worked was when I was using your API-Query which I did not recognize is from CodeDE.

Thanks again!
0 (0 Votes)

RE: Automatic Download (using python)?
Answer
22 June 2020 11:59 as a reply to Felix Keßler.
Hey Marcin,
I wanted to know if there is an update for my problem?

Best,
Felix
0 (0 Votes)

RE: Automatic Download (using python)?
Answer
25 June 2020 8:39 as a reply to Felix Keßler.
Hello,
Sorry for late reply - there's still no update. I've been encountering some technical issues on my side, will try to provide solution for your problem soon.

Regards,
Marcin
0 (0 Votes)

RE: Automatic Download (using python)?
Answer
26 June 2020 6:21 as a reply to Marcin Szreder.
Thanks for the No-Update-Update! Still happy to know that you at least think it is possible!
0 (0 Votes)

RE: Automatic Download (using python)?
Answer
22 July 2020 6:16 as a reply to Felix Keßler.
Hey Marcin,
I am still waiting for an update. Did you manage to find a solution by now?
0 (0 Votes)

RE: Automatic Download (using python)?
Answer
13 August 2020 9:34 as a reply to Felix Keßler.
Hi,
Sorry for late reply - unfortunately I couldn't find a solution for this problem.

Regards,
Marcin
0 (0 Votes)