Iota2 latest release – Deep Learning at the menu

For the experiments, we use all the Level-2A acquisitions available from Theia Data Center (https://www.theia-land.fr/en/products/) for one year (2018) over 4 Sentinel 2 tiles ([“31TCJ”, “31TCK”, “31TDJ”, “31TDK”]). See figure 1. The raw files size amount to 777 Gigabytes.

For the ground truth, we have extracted the data from the database used to produce the OSO product (https://www.theia-land.fr/en/ceslist/land-cover-sec/). The database was constructed by merging several open source databases, such as Corine Land Cover. The whole process is described in (Inglada et al. 2017). The 23-categories nomenclature is detailed here: https://www.theia-land.fr/en/product/land-cover-map/. An overview is given figure 2.

As usual with iota2, the first step is to set-up the configuration file. This file hosts most of the information required to run the computation (where are the data, the reference file, the output folder etc …). The following link is a good start to understand the configuration file: https://docs.iota2.net/develop/i2_classification_tutorial.html#understand-the-configuration-file. We try to make the following understandable without the need to fully read it.

To compute the classification accuracy obtained on the area covered by ground truth used for training, we indicate to iota2 to split polygons from the ground truth file into two files, one for training and one for testing with a ratio of 75%:

split_ground_truth : True
ratio : 0.75

It means that 75% of the available polygons for each class is used for training while the remaining is used for testing. Note that we do not talk about pixels here. By splitting at the polygons level, we ensure that pixels from a polygon are used either for training or testing. This is a way to reduce the spatial auto-correlation effect between pixels when assessing the classification accuracy.

We need now to set-up how training pixels are selected from the polygons. Iota2 relies on OTB sampling tools (https://www.orfeo-toolbox.org/CookBook/Applications/app_SampleSelection.html). For this experiment, we asked for a maximum number of pixels of 100000 with a periodic sampler.

arg_train :
{
    sample_selection :
  {
        "sampler" : "periodic"
        "strategy": "constant"
        "strategy.constant.nb" : 100000
  }

}

We are working on 4 different tiles, each of them having its own temporal sampling. Furthermore, we need to deals with clouds issues (Hagolle et al. 2010). Iota2 uses temporal gap-filling as discussed in (Inglada et al. 2015). In this work, we use a temporal step-size of 10 days, i.e., we have 37 dates. Iota2 also computes per default three indices (NDVI, NDWI and Brightness). Hence, for a each pixel we have a set of 481 features (37\(\times\)13).

For the deep neural network, we use the default implementation in iota2. However, it is possible to define its own architecture (https://docs.iota2.net/develop/deep_learning.html?highlight=deep%20learning#desc-dl-module). In our case, the network is composed of four layers (see Table 1) with a relu function between each of them (https://framagit.org/iota2-project/iota2/-/blob/develop/iota2/learning/pytorch/torch_nn_bank.py#L276).

ADAM solver was used for the optimization, with a learning_rate of \(10^{-5}\) and a batch size of 4096. 200 epochs were performed and a validation sample set, extracted from the training pixels is used to monitor the optimization. The best model in terms of Fscore is selected. Off course all these options are configurable with iota2. A full configuration file is given in Listing 2.

chain :
{
  output_path : "/datalocal1/share/fauvelm/blog_post_iota2_output/outputs_3"
  remove_output_path : True
  check_inputs : True
  list_tile : "T31TCJ T31TDJ T31TCK T31TDK"
  data_field : "code"
  s2_path : "/datalocal1/share/PARCELLE/S2/"
  ground_truth : "/home/fauvelm/BlogPostIota2/data/gt_3_clean.shp"

  spatial_resolution : 10
  color_table : "/home/fauvelm/BlogPostIota2/data/colorFile.txt"
  nomenclature_path : "/home/fauvelm/BlogPostIota2/data/nomenclature.txt"
  first_step : 'init'
  last_step : 'validation'
  proj : "EPSG:2154"
  split_ground_truth : True
  ratio : 0.75
}

arg_train :
{
  runs : 1
  random_seed : 0
  sample_selection :
  {
        "sampler" : "periodic"
        "strategy": "constant"
        "strategy.constant.nb" : 100000
  }
  deep_learning_parameters :
  {
        dl_name : "MLPClassifier"
        epochs : 200
        model_selection_criterion : "fscore"
        num_workers : 12
        hyperparameters_solver : {
                        "batch_size" : [4096],
                        "learning_rate" : [0.00001]
                        }
    }
}

arg_classification :
{
        enable_probability_map : True
}

python_data_managing :
{
        number_of_chunks : 50
}

sentinel_2 :
{
        temporal_resolution : 10
}

task_retry_limits :
{
         allowed_retry : 0
         maximum_ram : 180.0
         maximum_cpu : 40
}

The configuration file is now ready and the chain can be launched, as described in the documentation. Classification accuracy will be outputted in the directory final as well as the final classification map and related iota2 outputs.

However, in this post we want to go a bit further to enlighten how easy it is to run several simulations with iota2. As discussed in 2.1.2.1, we have generated a set of pair of spatially disjoint ground truth vector files for training and for testing. Also, remind that iota2 starts by splitting the provided training ground truth file into two spatially disjoints files, one used to train the model and the other used to test the model. In such particular configuration, we have now two test files:

1. One extracted from the same area than the training samples,
2. One extracted from a different area than the training samples.

With this files, we can do a spatial cross validation estimation of the classification accuracy, as discussed in (Ploton et al. 2020). To perform such analysis, we first stop the chain after the prediction of the classification map (setting the parameter as last_step : 'mosaic') and we manually add another step to estimate the confusion matrix from both sets. We rely on the OTB tools: https://www.orfeo-toolbox.org/CookBook/Applications/app_ComputeConfusionMatrix.html?highlight=confusion%20matrix. The last ingredient is to be able to loop over the different tiles configurations, i.e., to iterate over the cross-validation folds. This is where iota2 is really powerful: we just need to change few parameters in the configuration file to run all the different experiments. In this case, we have to change the ground truth filenames and the output directory. To make it simple, we indexed our simulations from 0 to 3 and use sed shell tool to modify the configuration file in the big loop:

sed -i "s/outputs_\([0-9]\)/outputs_${REGION}/" /home/fauvelm/BlogPostIota2/configs/config_base.cfg
sed -i "s/gt_\([0-9]\)_clean/gt_${REGION}_clean/" /home/fauvelm/BlogPostIota2/configs/config_base.cfg

The global script is given in Listing 3.

#/user/bin/bash

# Set ulmit
ulimit -u 6000

# Set conda env
source ~/.source_conda
conda activate iota2-env

# Loop over region
for REGION in 0 1 2 3
do
    echo Processing Region ${REGION}
    # (Delate and) Create output repertory
    OUTDIR=/datalocal1/share/fauvelm/blog_post_iota2_output/outputs_${REGION}/
    if [ -d "${OUTDIR}" ]; then rm -Rf ${OUTDIR}; fi
    mkdir ${OUTDIR}

    # Update config file
    sed -i "s/outputs_\([0-9]\)/outputs_${REGION}/" /home/fauvelm/BlogPostIota2/configs/config_base.cfg
    sed -i "s/gt_\([0-9]\)_clean/gt_${REGION}_clean/" /home/fauvelm/BlogPostIota2/configs/config_base.cfg

    # Run iota2
    Iota2.py \
        -config /home/fauvelm/BlogPostIota2/configs/config_base.cfg \
        -config_ressources /home/fauvelm/BlogPostIota2/configs/ressources.cfg \
        -scheduler_type localCluster \
        -nb_parallel_tasks 2


    # Compute Confusion Matrix for test samples
    otbcli_ComputeConfusionMatrix \
        -in ${OUTDIR}final/Classif_Seed_0.tif \
        -out ${OUTDIR}confu_test.txt \
        -format confusionmatrix \
        -ref vector -ref.vector.in /home/fauvelm/BlogPostIota2/data/tgt_0.shp \
        -ref.vector.field code \
        -ram 16384

    # Merge validation samples
    ogrmerge.py -f SQLITE -o ${OUTDIR}merged_val.sqlite \
                ${OUTDIR}dataAppVal/*_val.sqlite


    # Compute Confusion Matrix for train samples
    otbcli_ComputeConfusionMatrix \
        -in ${OUTDIR}final/Classif_Seed_0.tif \
        -out ${OUTDIR}confu_train.txt \
        -format confusionmatrix \
        -ref vector -ref.vector.in ${OUTDIR}merged_val.sqlite \
        -ref.vector.field code \
        -ram 16384
done

python compute_accuracy.py

Then we can compute classification metrics, such as the overall accuracy, the Kappa coefficient and the average Fscore. For this post, we have written a short python script to perform such operations: https://gitlab.cesbio.omp.eu/fauvelm/multitempblogspot/-/blob/main/scripts/compute_accuracy.py.

We can just run it, using nohup for instance, take a coffee, a slice of cheesecake and wait for the results 🙂

The figure 9 shows the Fscore for the 4 models (coming from the 4 different runs), and the 2 test sets. We can easily see that depending on the tile left out, the difference of classification accuracy in terms of Fscore between test samples extracted from the same or disjoint area can be significant. Discussing the reasons why the performance are decreasing and what metrics should we use to asses the map accuracy are out of the scope of this post. It is indeed a controversy topic in remote sensing (Wadoux et al. 2021). We just want to emphasize that iota2 simplifies and automatizes a lot the process of validation, especially at large scale. Using this spatial cross validation with 4 folds, the mean estimated Fscore is 0.59 with a standard deviation of 0.08, which is indeed much lower than the 0.73 estimated in the previous part.

The different classification maps are displayed in the animated figures 10, 11 and 12. The first one displays a tricky situation at a frontier of two tiles. We can see a strong discontinuity, whatever the model used. For the second case over Toulouse, there is a global agreement between the 4 models, except for the class Continuous Urban Fabric (pink) that disappears for one model: the one learnt without data coming from the tile containing Toulouse (T31TCJ). The last area exhibits a global agreement with some slight disagreements for some crops. Note, the objective is not to fuse/combine the different results, but rather to quantitatively observed the differences in terms of classification maps when the ground truth is changed.

A finer analysis could be done, indeed. But we let this as an homework for interested reader: all the materials for the simulation are available here

and the Level-2A MAJA processed Sentinel-2 data are downloadable from Theia Data Center (try this out: https://github.com/olivierhagolle/Sentinel-download), while the ground truth data can be downloaded here.