Forum

Message Boards Message Boards

Back

/eodata is cached to /1??

/eodata is cached to /1??
Answer
7 May 2021 6:26
I have recently switched from using boto3 based access to the S3 store to the NFS mounted /eodata because the recent upgrades showed that /eodata access is now as fast as boto3 based access. I am running parallel analysis with docker swarm which uses large time series of Sentinel-2 and -1.

My converted code that uses /eodata runs OK but apparently all the data that I read into memory (using VRT and rasterio) gets cached in directory /1/DIAS on my VM. Since that VM has only a moderate disk size, it gets stuffed after some time, after which everything blocks because I run out of disk space. I have not had this issue (with caching) when using boto3, because I make a temporary copy of the image data, process it and then delete it (i.e. using an explicit temprary cache).

Is this expected behavior? I have not found any pointers to this. I do not think it is a side-effect of using VRT but it may be related to my use of Numba.

Guido
0 (0 Votes)

RE: /eodata is cached to /1??
Answer
10 May 2021 5:43 as a reply to Guido Lemoine.
Hi Guido,

This effect happens when you use S3FS, not NFS.
See: https://creodias.eu/-/s3fs-cache?inheritRedirect=true&redirect=%2Ffaq-s3

Which image (name and version) have you used to spawn the VM?

Regards,
Marcin
+1 (1 Vote)

RE: /eodata is cached to /1??
Answer
11 May 2021 6:30 as a reply to Marcin Gil.
Thanks Marcin. I have indeed a s3fs mounted /eodata, with caching to /1. I now explicitly clear the cache. 
0 (0 Votes)