Speed-up downloads from PEPS S2 mirror site with peps_download.py

The French Sentinel mirror site, PEPS, has a very clever data management facility. All the products are stored on tapes, with a capacity of several PB, and there is some sort of cache made of disks. The products accessed recently are on disks, while the other products stay on tapes. The storage costs and also power consumption are therefore largely optimized. The drawback is that before accessing a file on tape, some time is needed to get the tape, and read the file on tapes. This can take something like 2 to 10 minutes. My little tool, peps_download.py was designed when most of the products were on disks, and it was quite slow to download products on tapes. As I am not a patient person, I have tried to speed it up, and it works well, thanks to good advise from CNES peps  colleagues (Christophe Taillan and Erwann Poupart). The previous version was working like that :

Make catalog requestFor all product in the request result :- while product is not downloaded - try to download the product - if still on tape, wait for 2 minutes

As a result, for each product on tape, it was necessary to wait for 2 to 10 minutes. Now, it works like that

Make catalog requestFor all products on tape in the request result- ask to read it on disksWhile (still some products to download):- Redo catalog request- Download products on disk- If some products are not on disk yet - wait for 2 minutes

 On my computer, it used to take more that 12 hours to download 2 years of Sentinel-2 data for a given tile. It has now been reduced to less that 3 hours (but my computer is on CNES network). I hope you will have similar results !

Plus d'actualités

Mo Chhu, dzong of Punakha.

Tracking the sediment plume of the 29 October 2026 GLOF in Bhutan

A glacial lake outburst flood (GLOF) occurred on 29 Oct in northern Bhutan, creating a debris flow in the Mo Chhu valley, Gasa district. The exact cause of the GLOF is still unclear but we know that it originates from Sintaphu Tsho, a moraine-dammed lake at an elevation of about 4500 m. This event has […]

Adjacency effects, or how pixels from the entire region can skew your measurements

=> Neighborhood issues also exist in Earth observation. And in a sense, they are even much more serious than on a social level, since neighbors located more than 30 km away can cause interference! Fortunately, we have the means to correct these problems without resorting to law enforcement—just using algorithms. This entire article is based […]

Les effets d’environnement, ou comment les pixels de toute la région peuvent perturber vos mesures.

=> Les problèmes de voisinage existent aussi en observation de la terre. Et en un sens, ils sont même beaucoup plus graves que sur le plan social, puisque vos voisins situés à plus de 30 km peuvent pour perturber ! Heureusement, nous disposons de moyens pour corriger ces problèmes, sans recourir à la force publique, […]

Rechercher