Speed-up downloads from PEPS S2 mirror site with peps_download.py

The French Sentinel mirror site, PEPS, has a very clever data management facility. All the products are stored on tapes, with a capacity of several PB, and there is some sort of cache made of disks. The products accessed recently are on disks, while the other products stay on tapes. The storage costs and also power consumption are therefore largely optimized. The drawback is that before accessing a file on tape, some time is needed to get the tape, and read the file on tapes. This can take something like 2 to 10 minutes. My little tool, peps_download.py was designed when most of the products were on disks, and it was quite slow to download products on tapes. As I am not a patient person, I have tried to speed it up, and it works well, thanks to good advise from CNES peps  colleagues (Christophe Taillan and Erwann Poupart). The previous version was working like that :

Make catalog requestFor all product in the request result :- while product is not downloaded - try to download the product - if still on tape, wait for 2 minutes

As a result, for each product on tape, it was necessary to wait for 2 to 10 minutes. Now, it works like that

Make catalog requestFor all products on tape in the request result- ask to read it on disksWhile (still some products to download):- Redo catalog request- Download products on disk- If some products are not on disk yet - wait for 2 minutes

 On my computer, it used to take more that 12 hours to download 2 years of Sentinel-2 data for a given tile. It has now been reduced to less that 3 hours (but my computer is on CNES network). I hope you will have similar results !

Plus d'actualités

Dates de fauche en France

En 2024, des collègues du Cesbio ont publié un article* sur la cartographie de la date de fauche en France en 2022 à partir des données satellitaires Sentinel-2.   Leur magnifique figure 10 a attiré mon attention. La version à haute résolution fournie avec l’article (2861 × 2911 pixels) correspond à une image dont les pixels font […]

Évolution du jour de déneigement dans les Alpes françaises et les Pyrénées

Les socio-écosystèmes des Alpes et des Pyrénées dépendent étroitement des fluctuations annuelles du manteau neigeux. En particulier, le moment de l’année où la neige disparait détermine le début de la saison de croissance de la végétation de montagne et donc la période des estives. Le changement climatique est en train de bouleverser ce rythme saisonnier. […]

Sentinel-2 reveals the surface deformation after the 2025 Myanmar earthquake

Sentinel-2 captured several clear-sky images of Myanmar before and after the 28 March 2025 earthquake. The animation below shows a 5-day apart sequence of images captured by Sentinel-2B and Sentinel-2C (10 m resolution) near the epicenter located close to Mandalay. The surface slip due to the earthquake follows the Sagaing Fault, a major fault in […]

Rechercher