Speed-up downloads from PEPS S2 mirror site with peps_download.py

The French Sentinel mirror site, PEPS, has a very clever data management facility. All the products are stored on tapes, with a capacity of several PB, and there is some sort of cache made of disks. The products accessed recently are on disks, while the other products stay on tapes. The storage costs and also power consumption are therefore largely optimized. The drawback is that before accessing a file on tape, some time is needed to get the tape, and read the file on tapes. This can take something like 2 to 10 minutes. My little tool, peps_download.py was designed when most of the products were on disks, and it was quite slow to download products on tapes. As I am not a patient person, I have tried to speed it up, and it works well, thanks to good advise from CNES peps  colleagues (Christophe Taillan and Erwann Poupart). The previous version was working like that :

Make catalog requestFor all product in the request result :- while product is not downloaded - try to download the product - if still on tape, wait for 2 minutes

As a result, for each product on tape, it was necessary to wait for 2 to 10 minutes. Now, it works like that

Make catalog requestFor all products on tape in the request result- ask to read it on disksWhile (still some products to download):- Redo catalog request- Download products on disk- If some products are not on disk yet - wait for 2 minutes

 On my computer, it used to take more that 12 hours to download 2 years of Sentinel-2 data for a given tile. It has now been reduced to less that 3 hours (but my computer is on CNES network). I hope you will have similar results !