[MUSCATE & THEIA] Production of Sentinel-2 L2A is late

Update on 25 september 2023 : we have no backlog left in Europe, and the system is catching up on the other regions of the world

Update on 9 september 2023 : 25 days processed in a week over France ! We are catching up a bit

CNES has bought and built a new HPC cluster, named T-Rex, that will soon replace the former one, HAL. T-Rex will drastically improve our processing capacity ! T-Rex started its operations this summer, but the transition is complex, as T-Rex reuses the most powerful nodes of HAL, needs a synchronisation of all the data sets (the data volumes to copy are huge), and of course, has a different OS version and a new scheduler. We have anticipated the changes in our systems, using a simulated environment to test our softwares, but, you know, simulations are not the reality.

As a result, the production of MUSCATE (THEIA) is still running on HAL, but some of the best processors of HAL have been migrated to T-REX, reducing our production capacity. moreover, we have been asked to stop producing on week-ends, to allow a faster copy of the data from HAL to T-Rex.

As a result, yet, we have not been able to catch-up the production that we had halted for a few weeks when the new version of CAMS was put in production, and for some sites, for instance, in France, we are late by one month. Please be assured that the teams are doing their best to catch it up.