Quantitative comparison of cloud masks from MACCS/MAJA, Sen2Cor and GEOSYS (hand made)

=> As already explained in a previous post, we obtained some Sentinel-2 hand made cloud masks from GEOSYS company. We used those to validate the cloud masks from MACCS/MAJA. But we wanted to use them further to make a quantitative comparison with Sen2Cor cloud masks.But this comparison required solving a little issue : GEOSYS cloud masks are generously dilated to avoid any risk to let a cloud pass through the operational processing. Those of MACCS:MAJA are also dilated while those of Sentinel-2 are not at all. In the following paragraphs, we’ll explain how we solved that issue. Sen2cor (v2.3.0) has also three levels of cloud mask (High Medium and Low probability). We used here the Medium Probability mask. But let’s start with the final result comparing the performances of Sen2Cor and MACCS:MAJA. 

Overall accuracies for MACCS/MAJA, in red and Sen2cor, in blue for 11 images compared to GEOSYS cloud masks.

Direct comparison of Sen2Cor et GEOSYS masks :

SEN2COR/GEOSYS
20161231confusion %cloud_sen2corclear_sen2corOA
cloud_geosys14.689.67
clear_geosys3.3672.2986.97
20170103confusion %cloud_sen2corclear_sen2corOA
cloud_geosys0.714.99
clear_geosys3.3390.9891.69
20170113confusion %cloud_sen2corclear_sen2corOA
cloud_geosys15.7420.26
clear_geosys2.361.6977.43
20170202confusion %cloud_sen2corclear_sen2corOA
cloud_geosys60.6326.31
clear_geosys0.9912.0772.7
20160209confusion %cloud_sen2corclear_sen2corOA
cloud_geosys81.8911.36
clear_geosys0.136.6288.51

The overall accuracy (OA), which is the percentage of well classified pixel, is not very good,  85%. We noted that when there are many small clouds in the image, there were many pixels classified as cloud free by Sen2Cor, which are classified as clouds by GEOSYS. Some part of teh difference might be due to the difference in dilation.

Comparison of SEN2COR dilated cloud mask with GEOSYS cloud mask

To solve this issue, I dilated the Sen2Cor cloud mask by the same buffer as MACCS cloud mask (480m).The results are shown hereafter :

SEN2COR dilaté /GEOSYS
20161231confusion %cloud_sen2corclear_sen2corOA
cloud_geosys22.911.43
clear_geosys38.637.0659.97
20170103confusion %cloud_sen2corclear_sen2corOA
cloud_geosys3.522.17
clear_geosys43.9752.5156.03
20170113confusion %cloud_sen2corclear_sen2corOA
cloud_geosys31.474.53
clear_geosys29.3834.6266.09
20170202confusion %cloud_sen2corclear_sen2corOA
cloud_geosys80.586.36
clear_geosys8.114.9585.53
20160209confusion %cloud_sen2corclear_sen2corOA
cloud_geosys88.744.51
clear_geosys2.374.3893.12

 We see that results improve for the mos cloudy images, but decrease a lot for the most cloud free images. This is due to the fact that Sen2Cor often wrongly detects buildings as clouds. Once dilates, these small clouds cover a large surface, as in the image below, and degrade the comparison.

Comparison of Sen2cor cloud mask, before dilation, with MACCS cloud mask, over Orléans tile (31UDP). Clouds are circled in green and shadows in yellow.

Comparison of Sen2Cor et GEOSYS eroded cloud mask

To miitigate this issue again, instead of dilating Sen2Cor cloud mask, I eroded GEOSYS cloud mask, before comparing it to Sen2Cor. Here are the results for the same images over Orléans :

SEN2COR/GEOSYS Erodé
20161231confusion %cloud_sen2corclear_sen2corOA
cloud_geosys14.548.29
clear_geosys3.5173.6788.21
20170103confusion %cloud_sen2corclear_sen2corOA
cloud_geosys0.653.87
clear_geosys3.3992.0992.74
20170113confusion %cloud_sen2corclear_sen2corOA
cloud_geosys15.0414.8
clear_geosys367.1682.2
20170202confusion %cloud_sen2corclear_sen2corOA
cloud_geosys59.8424.62
clear_geosys1.7813.7673.6
20160209confusion %cloud_sen2corclear_sen2corOA
cloud_geosys81.3210.54
clear_geosys0.77.4488.76

 In tha case, we get some improvement comparerd the the direct comparison. The large number of false detection covers only a little percentage of the image, and the agreement with GEOSYS reference is correct for the least cloudy images. But large discrepancies are still observed for the very cloudy images. Anyway, you know how kind I am, i selected the result most favorable to Sen2Cor to make the comparison with MACCS/MAJA, which I extended to the three available sites in France (Arles, Orleans et Toulouse); 

Comparison with MACCS/MAJA

If we summarize, the overall accuracy of SEN2COR or MACCS / MAJA cloud masks using GEOSYS (eroded in the case of SEN2COR) as reference, the results already presented above are obtained. It can be seen that MACCS / MAJA prevails in all but one case (Orléans 20170215), with a very low difference on that case. In general, the two codes are more or less in agreement when the images are clear, but diverge when the images are cloudy. The average overall accuracy for all cases is 94.2 % for MACCS/MAJA and 86.1 % for SEN2CORAs a side result, this study confirms that it is also rather complicated to dilate Sen2Cor cloud because of the scattered wrong clouds. The fact that the reference cloud masks include some dilatation might be a cause of the difference, but anyway, the results are a clear demonstration that over France, MACCS/MAJA cloud masks are fit for an operational use.  Encore un grand merci à GEOSYS pour avoir fourni ses masques de référence, et à l’ESA pour la distribution du logiciel Sen2Cor (sans oublier les données Sentinel-2A)  

Plus d'actualités

BIOMASS, the third launched satellite mission designed at CESBIO !

After SMOS in 2009, and VENµS in 2017, the CESBIO Laboratory is very proud to see its third proposed mission, Biomass, reach orbit. As always, it has been a long journey from the idea, at the beginning of the century, to the selection in 2013 as the seventh Earth Explorer Mission by ESA, to the […]

Sentinel-2 reveals the surface deformation after the 2025 Myanmar earthquake

Sentinel-2 captured several clear-sky images of Myanmar before and after the 28 March 2025 earthquake. The animation below shows a 5-day apart sequence of images captured by Sentinel-2B and Sentinel-2C (10 m resolution) near the epicenter located close to Mandalay. The surface slip due to the earthquake follows the Sagaing Fault, a major fault in […]

Evolution de l’altitude de la ligne de neige au cours des 41 dernières années dans le bassin versant du Vénéon (Oisans)

Pour contribuer à caractériser les conditions hydrométéorologiques lors de la crue torrentielle qui a frappé la Bérarde en juin, j’ai analysé une nouvelle série de cartes d’enneigement qui couvre la période 1984-2024 [1]. Grâce à la profondeur temporelle de cette série, on constate que l’altitude de la ligne de neige dans le bassin versant du […]

Rechercher