Speed-up downloads from PEPS S2 mirror site with peps_download.py

Code PEPS Sentinel-1 Sentinel-2

Olivier Hagolle, 23 mars 2018

The French Sentinel mirror site, PEPS, has a very clever data management facility. All the products are stored on tapes, with a capacity of several PB, and there is some sort of cache made of disks. The products accessed recently are on disks, while the other products stay on tapes. The storage costs and also power consumption are therefore largely optimized. The drawback is that before accessing a file on tape, some time is needed to get the tape, and read the file on tapes. This can take something like 2 to 10 minutes. My little tool, peps_download.py was designed when most of the products were on disks, and it was quite slow to download products on tapes. As I am not a patient person, I have tried to speed it up, and it works well, thanks to good advise from CNES peps  colleagues (Christophe Taillan and Erwann Poupart). The previous version was working like that :

Make catalog requestFor all product in the request result :- while product is not downloaded - try to download the product - if still on tape, wait for 2 minutes

As a result, for each product on tape, it was necessary to wait for 2 to 10 minutes. Now, it works like that

Make catalog requestFor all products on tape in the request result- ask to read it on disksWhile (still some products to download):- Redo catalog request- Download products on disk- If some products are not on disk yet - wait for 2 minutes

 On my computer, it used to take more that 12 hours to download 2 years of Sentinel-2 data for a given tile. It has now been reduced to less that 3 hours (but my computer is on CNES network). I hope you will have similar results !

Plus d'actualités

Sentinel-2 reveals the surface deformation after the 2025 Myanmar earthquake

Sentinel-2 captured several clear-sky images of Myanmar before and after the 28 March 2025 earthquake. The animation below shows a 5-day apart sequence of images captured by Sentinel-2B and Sentinel-2C (10 m resolution) near the epicenter located close to Mandalay. The surface slip due to the earthquake follows the Sagaing Fault, a major fault in […]

Evolution de l’altitude de la ligne de neige au cours des 41 dernières années dans le bassin versant du Vénéon (Oisans)

Pour contribuer à caractériser les conditions hydrométéorologiques lors de la crue torrentielle qui a frappé la Bérarde en juin, j’ai analysé une nouvelle série de cartes d’enneigement qui couvre la période 1984-2024 [1]. Grâce à la profondeur temporelle de cette série, on constate que l’altitude de la ligne de neige dans le bassin versant du […]

Biophysical parameter retrieval from Sentinel-2 images using physics-driven deep learning for PROSAIL inversion

The results presented here are based on published work: Y. Zérah, S. Valero, and J. Inglada. « Physics-constrained deep learning for biophysical parameter retrieval from sentinel-2 images: Inversion of the prosail model« , in Remote Sensing of Environment, doi: 10.1016/j.rse.2024.114309. This work is part of the PhD of Yoël Zérah, supervised by Jordi Inglada and Silvia Valero. […]

Rechercher