Sentinel-2 Enhance button: 5-meters resolution for 10 bands at your fingertips

Enhance button is a very common movies trope, where a character scrolls through some video footage or photos and asks a computer to enhance its resolution to an insane level of details, enabling solving crime mysteries and conspiracies of all sort with clues that were invisible in the original image. While this meme has been frequently parodied and mocked for defiling science, it did not stop researchers and engineers of our deep-learning era to harness the power of both GPUs and automatic differentiation into pushing foward performances of Single Image Super-Resolution (SISR), which is the scientific name of the Enhance button.

Of course, the remote sensing community wants its enhance button too! In the frame of the EVOLAND project, CESBIO has developed and released under Apache 2.0 sentinel2_superresolution, a tool that takes as input a L1C or L2A (Theia format) Sentinel-2 product and outputs 10 of the most useful bands (namely B2, B3, B4, B5, B6, B7, B8, B8A, B11, B12) at 5-meter resolution. It is very simple to use:

$ sentinel2_superesolution -v -i SENTINEL2A_20200808-105933-164_L2A_T31TCG_C_V2-2/ -o results/ -roi 300160.000 4590400.000 304000.000 4594240.000

Et voilà !

Backstory

Here at CESBIO, we released in 2022 the Sen2Venµs dataset, tailored for the training of SISR models that bring common bands of Venµs and Sentinel-2 at 5-meter resolution, that we gathered during a phase-0 study for the Sentinel-HR mission (add link). In 2022, we then worked together with Thales and MEOSS in the frame of one year ITT project for ESA, starting to explore the possibilities of the Sen2Venµs dataset for the super-resolution of Sentinel-2 with good performances. In 2023, we then joined the consortium of the EVOLAND project, a HORIZON Europe project from the European Commission aiming at designing future and enhanced Copernicus products for 2030, based on new missions, new data and new algorithms. CESBIO is responsible for the improved resolution sub-task, aiming at providing methods to improve the spatial, temporal and spectral resolution of satellite data for downstream product prototypes.

How it works

If SISR is to be part of the processing of future Copernicus products, it will be applied to tenths of thousands of products, and therefore the computational cost should be as limited as possible. We therefore selected a lightweight and relatively shallow network known as Cascading Residual Network (CARN), and parameterized it with as few blocks as possible in order to reduce its processing cost (selected model only has 2.5M parameters).The network has been trained  to jointly process all 10 bands at once, up-sampled at 10-meter resolution. Training is achieved by using the Sen2Venµs dataset complemented with B11 and B12 patches. In order to workaround the lack of 5-meter Venµs reference for  B11 and B12, training makes use of two loss terms, one at full resolution for all bands except B11 and B12, and the other specifically targeted at B11 and B12 and operating at lower resolution through Wald protocol.

The network is pre-trained with simulated data (from Venµs reference patches) for 20 epochs. We found that this pre-training already gives consistent results with respect to pre-training on real data, as shown in the following figure.

The best network from pre-training is then fine-tuned with adversarial (GAN) training for 10 additional epochs while monitoring the BRISQUE score on the validation set to select best model. This fine-tuning results in minor quality improvement, interestingly mostly benefiting the 20-meter bands, as shown in the following figure:

Performances assessment

Assessing performances of SISR network trained with the Sen2Venµs dataset is a challenging task, as already identified in the earlier work with Thalès and MEOSS, because the dataset has a lot of residual geometric and radiometric discrepancies between both satellite images which impairs traditional IQ metrics such as PSNR and SSIM. We first measure the radiometric consistency of each band with respect to the input Sentinel-2 image, over our testing set. Here we can observe that all bands have a RMSE below 0.005 for the training on simulated data (red bars), whereas training with L1 loss on real data or even with the more advanced HR/LR loss incur radiometric distortion.

The next and more difficult question is how much high resolution details are actually injected by the algorithm. The first thing we can do is to measure the RMSE on high spatial frequency content of the signal. Since this measure will be very sensitive to geometric distortion, we do it on simulated data. This highlights a moderate improvement for 10-meter bands and a large improvement for 20-meter band, which is consistent with the visual assessment.

Finally we can have a look at what happens in Fourier domain. We can observe that the super-resolved image populates Fourier domain in twice the extent of the initial signal, which shows that super-resolution actually restores higher spatial frequencies with respect to bicubic up-sampling.

Push the enhance button!

The sentinel2_superresolution can easily be installed with pip and should be straightforward to use. We have made the following estimates for processing time for a whole Sentinel-2 product, which is even achievable without a GPU !

In the frame of EVOLAND, we are also working on getting the Sentinel-2 super-resolution algorithm running in OpenEO, and open source code for that should be released very soon. Stay tuned!

We are looking forward to see how this tool will be used by the downstream product prototypes of EVOLAND. Along with Thalès and MEOSS, we already demonstrated a clear interest for the Water Bodies Detection task, especially with the super-resolution of the SWIR bands (B11 and B12). We are also planning to integrate the tool as an on-demand processing in GEODES. Feedback is also welcome.

Credits

This work was partly performed using HPC resources from GENCI-IDRIS (Grant 2023-AD010114835)

This work was partly performed using HPC resources from CNES.