Quantitative comparison of cloud masks from MACCS/MAJA, Sen2Cor and GEOSYS (hand made)

=> As already explained in a previous post, we obtained some Sentinel-2 hand made cloud masks from GEOSYS company. We used those to validate the cloud masks from MACCS/MAJA. But we wanted to use them further to make a quantitative comparison with Sen2Cor cloud masks.But this comparison required solving a little issue : GEOSYS cloud masks are generously dilated to avoid any risk to let a cloud pass through the operational processing. Those of MACCS:MAJA are also dilated while those of Sentinel-2 are not at all. In the following paragraphs, we’ll explain how we solved that issue. Sen2cor (v2.3.0) has also three levels of cloud mask (High Medium and Low probability). We used here the Medium Probability mask. But let’s start with the final result comparing the performances of Sen2Cor and MACCS:MAJA. 

Overall accuracies for MACCS/MAJA, in red and Sen2cor, in blue for 11 images compared to GEOSYS cloud masks.

Direct comparison of Sen2Cor et GEOSYS masks :

The overall accuracy (OA), which is the percentage of well classified pixel, is not very good,  85%. We noted that when there are many small clouds in the image, there were many pixels classified as cloud free by Sen2Cor, which are classified as clouds by GEOSYS. Some part of teh difference might be due to the difference in dilation.

Comparison of SEN2COR dilated cloud mask with GEOSYS cloud mask

To solve this issue, I dilated the Sen2Cor cloud mask by the same buffer as MACCS cloud mask (480m).The results are shown hereafter :

 We see that results improve for the mos cloudy images, but decrease a lot for the most cloud free images. This is due to the fact that Sen2Cor often wrongly detects buildings as clouds. Once dilates, these small clouds cover a large surface, as in the image below, and degrade the comparison.

Comparison of Sen2cor cloud mask, before dilation, with MACCS cloud mask, over Orléans tile (31UDP). Clouds are circled in green and shadows in yellow.

Comparison of Sen2Cor et GEOSYS eroded cloud mask

To miitigate this issue again, instead of dilating Sen2Cor cloud mask, I eroded GEOSYS cloud mask, before comparing it to Sen2Cor. Here are the results for the same images over Orléans :

 In tha case, we get some improvement comparerd the the direct comparison. The large number of false detection covers only a little percentage of the image, and the agreement with GEOSYS reference is correct for the least cloudy images. But large discrepancies are still observed for the very cloudy images. Anyway, you know how kind I am, i selected the result most favorable to Sen2Cor to make the comparison with MACCS/MAJA, which I extended to the three available sites in France (Arles, Orleans et Toulouse); 

Comparison with MACCS/MAJA

If we summarize, the overall accuracy of SEN2COR or MACCS / MAJA cloud masks using GEOSYS (eroded in the case of SEN2COR) as reference, the results already presented above are obtained. It can be seen that MACCS / MAJA prevails in all but one case (Orléans 20170215), with a very low difference on that case. In general, the two codes are more or less in agreement when the images are clear, but diverge when the images are cloudy. The average overall accuracy for all cases is 94.2 % for MACCS/MAJA and 86.1 % for SEN2CORAs a side result, this study confirms that it is also rather complicated to dilate Sen2Cor cloud because of the scattered wrong clouds. The fact that the reference cloud masks include some dilatation might be a cause of the difference, but anyway, the results are a clear demonstration that over France, MACCS/MAJA cloud masks are fit for an operational use.  Encore un grand merci à GEOSYS pour avoir fourni ses masques de référence, et à l’ESA pour la distribution du logiciel Sen2Cor (sans oublier les données Sentinel-2A)