Quantitative validation of MACCS cloud masks

=> At the annual review  of PEPS project (CNES Sentinel global mirror site), the GEOSYS company showed its operational activities centered on the use of Sentinel-2 data to provide advise on agriculture on many regions of the world. On this occasion, GEOSYS showed their cloud detection process for Sentinel-2 images. The Sen2cor solution was not considered reliable enough y GEOSYS, and the regions processed by MACCS within MUSCATE are far from covering all the regions of interest of the company. GEOSYS decided to rely on human operators to improve the cloud masking. For each processed Sentinel-2 image, a man made valid pixel mask is build (« valid » means without clouds and cloud shadows). 

I immediately jumped at the bait and suggested a collaboration to GEOSYS, to compare MACCS valid pixel masks to the manual classification of GEOSYS. In this framework, GEOSYS kindly gave me access to several of its cloud masks that I just compared to those from MACCS. Data were processed over 4 tiles scattered over France, during a 3 months period from December 2016 to February 2017 :

 The tile from Rennes in Britanny did not provide resuts, all the images are fully cloudy and MACCS does not issue a product when the cloud cover is above 90%. But we obtained very good results over the three other sites which are displayed just below :

 This table displays all the confusion matrices obtained for each processed image, and the OA (Overall Accuracy) column provides the percentages of well-classified pixels. This rate goes from 86 to 99.5%. The agreement is excellent, but of course, the worse results bring us more information on how to enhance our method. You will find below two examples of disagreements, please click on the images to see them at full resolution, and see the legends for interpretation.

As a conclusion of this first independent quantitative validation of our Sentinel2 products, GEOSYS and MACCS agree on 94.5% on average, on 11 different images. The GEOSYS operators do a great work and can detect even the faintest clouds. However, they also tend to dilate the broken clouds generously. Indeed, GEOSYS needs to absolutely avoid cloud omissions, and, besides, the productivity is increased by simplifying the polygons instead of drawing a polygon around each single cloud.  MACCS clouds are dilated too, but not as much as those of GEOSYS. In some cases, MACCS may also miss the faintest clouds, and also thicker ones with a very small surface. We would like to increase the resolution of our cloud mask but it would be at the expense of computation time.

These results will be completed by comparison to other processors. The results will be shown on this blog of course, as well as the RAQRS V symposium in Valencia in September.

Many thanks to Arnaud Quesney (from GEOSYS) and GEOSYS for providing the data and helping me writing this post.