Your Processing Environment

Searching, processing and analysis of Sentinel-5P data on CREODIAS


On the CREODIAS the full archive of the Sentinel-5P mission is hosted. The Sentinel-5P satellite, with onboard the TROPOMI sensor, is designed as a precursor satellite, to fill in the data gap and provide data continuity between the retirement of the Envisat satellite and NASA's Aura mission and the launch of Sentinel-5. The mission performs atmospheric monitoring and was launched in October 2017. By accessing the Sentinel-5P data repository through a CREODIAS Virtual Machine, users can process data without any need for downloading data. In this case study, Sentinel-5P absorbing aerosol index (AAI) data is collected for India in October and November 2019 with the CREODIAS Finder API. Subsequently, the data is pre-processed with HARP tools and results are visualised, all on a CREODIAS VM.

The Absorbing Aerosol Index (AAI) indicates the presence of elevated absorbing aerosols in the Earth's atmosphere. The aerosol types that are mostly seen in the AAI are desert dust and biomass burning aerosols. In the extreme air pollution suffered in India in October and November 2019, AAI values of >3 have been recorded.

Sentinel-5P data access

For this use case, a Windows VM on CREODIAS has been used. On the VM, all the Sentinel-5P data is mounted as a network drive that can be accessed through File Explorer and directly loaded in software like ESA SNAP.

The data catalogue can be queried through the CREODIAS Finder or Finder API. Through the API, a query can be submitted through Python or any other scripting language. The returned information is in either JSON, ATOM format, or just the path names of the files that match the query.

For this use case, a Python script is used to send an API query and parse the results into a data frame. From this data frame the column with path names are extracted to be used for the processing. Using the CREODIAS Finder API is free of charge.

Figure 1: Screendump API query Python script

Processing Sentinel-5P data

For the processing of Sentinel-5P, the HARP tools are used. This extensive Python library is developed by Science [&] Technology Corporation (S[&]T) to process and analyse a wide range of atmospheric EO data sets and is well-equipped to deal with Sentinel-5P data.

The path names to the Sentinel-5P aerosol images that were returned in the Finder API query are ingested in the HARP software, image data that does not meet a quality threshold are filtered out and the remaining data is combined into a daily average. This can be done for any Sentinel-5P dataset, for any area and for any date range. The resulting data is converted to a geotiff for further analysis. Besides the automated Python scripting, there is also a GUI available to use the HARP tools in a more interactive way, named VISAN. With this tool, Sentinel-5P data can be checked and plotted easily.

Figure 2 Example of Sentinel-5P NO2 data visualised in VISAN

Visualisation of daily aerosol data

The daily averages of aerosol optical depth over India for the months October/November 2019 were prepared with HARP. AAI raster images were visualised with mapping libraries in R into daily maps. These were stacked into an animated gif file, to show the development of AAI over two months. It is clear from this image that the peak of AAI occurred around late October 2019.

Figure 3 Map of Absorbing Aerosol Index over India on 31/10/2019

Use potential

This use case shows how a CREODIAS VM allows a user to search, process and visualise Sentinel-5P data without any need to download data. This provides opportunity to use these datasets storage and computationally efficient, without any need for large storage and processing facilities on premise. The scripts used in this use case are simple and can be adapted to any other Sentinel-5P data product, time range, area as wished.


Full copies of the Sentinel-5P data sets are stored on CREODIAS. Using a VM on CREODIAS allows a user to search, process, analyse and visualise data through simple scripts, through open source Python and R libraries.