Running MAJA with startmaja

Startmaja is intended to help users to launch the processing of a time series of products by:

  • Downloading the adequate configuration files (ie. the GIPP collection)
  • Downloading and preparing the Digital Terrain Model that matches the tile of the L1C product
  • Explore available Level-1C products to define an appropriate workplan

Typically, if you have a collection of 20 Sentinel-2A L1C products for a given tile matching the date interval of interest, startmaja will automatically define a workplan that includes:

  • The processing of the first 8 L1C products in backward mode to ensure proper initialisation of the first (oldest) product
  • The processing of the entire series of L1C products in nominal (forward) mode

Startmaja can process time-series for one tile at a time. If you have a collection that includes many tiles, you’ll have to launch one startmaja job per tile.

Requirements

To launch startmaja, you will need:

  • A directory with L1C products. We recommend you organize your data directory by site (area of interest), then by tile. As an example, ~/data/Avignon/T31TFJ/
  • A Digital Terrain Model (DTM). If you wish to prepare the DTM by yourself prior to launch startmaja, you may use the dtmcreation utility available in the ../bin directory. Otherwise, startmaja will attempt to download and prepare it for you the first time you process a given tile. DTMs are stored by tile id in a ../dtm directory
  • [optional] If you wish to use the auxiliary Copernicus Atmosphere Monitoring Service (CAMS) aerosols, you’ll need to download them using the camsdownload utility available in the ../bin directory.
  • The path to the input, auxiliary and output data, all defined in a file named ‘folder.txt’ that you can create at the root location of your work directory. It’s content should match the following:
[Maja_Inputs]
repWork=./work
repGipp=./gipp
repMNT=./dtm
repL1  =/path/to/L1C
repL2  =/path/to/L2A
exeMaja=/path/to/bin/maja
repCAMS=/path/to/CAMS

[DTM_Creation]
repRAW=./dtm/raw 
repGSW=./dtm/gsw

Description of ‘folder.txt’:

  • repWork is a directory to store the temporary files. It should be a different location from your working directory (eg. ~/tmp)
  • repGipp is the folder where startmaja automatically downloads the GIPP for each configuration
  • repMNT stores the DTM (MNT in french) in Maja format, generated by the dtmcreation tool
  • repL1 is where to find the L1C data (without the site name which is added afterward optionally)
  • repL2 is for the L2A data (without the site name which is added afterwards, optionally again)
  • exeMaja is the full path to MAJA’s executable file
  • repCAMS is where CAMS data is stored. You do not need to specify this directory if you decide to not process with CAMS option
  • repRAW stores the raw DTM archives (such as the ones for SRTM, which have the name srtm_37_04.zip)
  • repGSW stores the raw Water-Mask files (such as the one for GSW, which have the name occurrence_0E_50N*.tif)

Launch startmaja

usage: startmaja [-h] -t TILE [-s SITE] -f FOLDER [-d START] [-e END] [-v]
                     [--nbackward NBACKWARD] [--overwrite] [--cams]
                     [--type_dem {srtm,eudem,glo30,any}] [-y] [--skip_errors]
                     [--version] [--platform {sentinel2,landsat8,venus}]
                     [--output_format {natif,muscate}] [--userconf USERCONF]
                     [--max_l2_diff MAX_L2_DIFF] [--ChangeTargetResToR3]

Description of command line options:

  • -h, –help : show docstring
  • -t TILE, –tile TILE : tile ID, eg. 29RPQ
  • -s SITE, –site SITE : site name, which must match the name of the parent directory of the tile directory
  • -f FOLDER, –folder FOLDER : name of the folder definition file, eg. ‘folder.txt’
  • -d START, –start START : start date of the period to be processing, with format YYYY-MM-DD. If undefined, all the L1C found in will be processed
  • -e END, –end END : end date of the period to be processing, with format YYYY-MM-DD
  • -v, –verbose : provides detailed (DEBUG) logging for Maja
  • –nbackward NBACKWARD : number of products used in backward mode, defaults to 8
  • –overwrite : overwrite any pre-existing L2A product in
  • –cams : use CAMS auxiliary data. Assumes products availability in directory
  • –type_dem {srtm,eudem,glo30,any} : define which DTM source to use in no DTM found
  • -y : skip workplan confirmation (mandatory if you use startmaja on HPC with a job submission tool such as PBDPro or slurm)
  • –skip_errors : keep processing according to the workplan even if one product falls down in error
  • –platform {sentinel2,landsat8,venus} : select sensor GIPP to be used, though detected from the L1C meta-data by default. We recommend you keep the default behavior
  • –max_l2_diff MAX_L2_DIFF : maximum interval in days between an L1C and a pre-existing L2A. If larger than MAX_L2_DIFF, MAJA will restart in backward. Otherwise MAJA will restart from the latest L2A. We recommend you keep the default behavior

Example:

../maja/4.x/bin/startmaja -f folders.txt -t 30TYQ -s Gers -d 2022-08-01 -e 2022-09-15 --cams -y

Note:

  • To fully benefit from the multi-temporal capabilities of MAJA, it is recommended to provide a series of L1C products. The default number of L1C needed for backward mode is 8 L1C products;
  • If you only provide one L1C product, MAJA will run in ‘init’ mode, using only multi-spectral criteria. It is not the recommended way of using MAJA;
  • When a L1C product has more than 90% cloud coverage, the L2A is not issued. This results in the output folders to contain only a metadata (MTD*xml) file but no rasters/images.

Commented example of workplan

Here is the typical output you’d get by launching startmaja to process a bunch of 18 Sentinel-2 L1C products from tile ID ’30TYQ’ from site named ‘Gers’, with the use of CAMS auxiliary data. Comments are provided below.

2023-01-06 13:25:03,189 [INFO ] =============This is Start_Maja v4.6.0==============
2023-01-06 13:25:08,108 [WARNI] 30TYQ
2023-01-06 13:25:22,199 [INFO ] Detecting input products...
2023-01-06 13:25:22,247 [INFO ] 18 L1C product(s) detected for site Gers and tile 30TYQ in /work/CESBIO/projects/Maja/L1C/Gers/T30TYQ
2023-01-06 13:25:22,247 [WARNI] No L2A products detected for site Gers and tile 30TYQ in /work/CESBIO/projects/Maja/L2A_MAJA/Gers/30TYQ
2023-01-06 13:25:22,248 [INFO ] Searching for CAMS
2023-01-06 13:26:18,562 [INFO ] ...found 558 CAMS files
2023-01-06 13:26:18,563 [INFO ] Checking GIPP files
2023-01-06 13:26:18,564 [INFO ] Setting up GIPP folder: /work/CESBIO/users/toto/runs_460/maja-gipp2
2023-01-06 13:26:19,075 [INFO ] Searching for DTM
2023-01-06 13:26:19,094 [INFO ] Found DTM: /work/CESBIO/users/toto/runs/DTM/S2__TEST_AUX_REFDE2_30TYQ_1001.HDR
2023-01-06 13:26:19,626 [INFO ] GIPP Creation succeeded for SENTINEL2
2023-01-06 13:26:19,889 [INFO ] 18 workplan(s) successfully created:
               DATE |       TILE |     MODE |                                                             L1-PRODUCT | ADDITIONAL INFO
2022-08-01 10:56:31 |      30TYQ | BACKWARD |      S2A_MSIL1C_20220801T105631_N0400_R094_T30TYQ_20220801T144009.SAFE | Backward of 8 products with CAMS
2022-08-03 10:46:29 |      30TYQ |  NOMINAL |      S2B_MSIL1C_20220803T104629_N0400_R051_T30TYQ_20220803T113612.SAFE | L2 from previous with CAMS
2022-08-06 10:56:29 |      30TYQ |  NOMINAL |      S2B_MSIL1C_20220806T105629_N0400_R094_T30TYQ_20220806T114519.SAFE | L2 from previous with CAMS
2022-08-08 10:46:31 |      30TYQ |  NOMINAL |      S2A_MSIL1C_20220808T104631_N0400_R051_T30TYQ_20220808T142930.SAFE | L2 from previous with CAMS
2022-08-11 10:56:31 |      30TYQ |  NOMINAL |      S2A_MSIL1C_20220811T105631_N0400_R094_T30TYQ_20220811T162332.SAFE | L2 from previous with CAMS
2022-08-13 10:46:29 |      30TYQ |  NOMINAL |      S2B_MSIL1C_20220813T104629_N0400_R051_T30TYQ_20220813T113233.SAFE | L2 from previous with CAMS
2022-08-16 10:56:19 |      30TYQ |  NOMINAL |      S2B_MSIL1C_20220816T105619_N0400_R094_T30TYQ_20220816T114611.SAFE | L2 from previous with CAMS
2022-08-18 10:46:31 |      30TYQ |  NOMINAL |      S2A_MSIL1C_20220818T104631_N0400_R051_T30TYQ_20220818T161035.SAFE | L2 from previous with CAMS
2022-08-21 10:56:31 |      30TYQ |  NOMINAL |      S2A_MSIL1C_20220821T105631_N0400_R094_T30TYQ_20220821T144116.SAFE | L2 from previous with CAMS
2022-08-23 10:46:19 |      30TYQ |  NOMINAL |      S2B_MSIL1C_20220823T104619_N0400_R051_T30TYQ_20220823T125620.SAFE | L2 from previous with CAMS
2022-08-26 10:56:19 |      30TYQ |  NOMINAL |      S2B_MSIL1C_20220826T105619_N0400_R094_T30TYQ_20220826T114739.SAFE | L2 from previous with CAMS
2022-08-28 10:46:31 |      30TYQ |  NOMINAL |      S2A_MSIL1C_20220828T104631_N0400_R051_T30TYQ_20220828T143127.SAFE | L2 from previous with CAMS
2022-08-31 10:56:31 |      30TYQ |  NOMINAL |      S2A_MSIL1C_20220831T105631_N0400_R094_T30TYQ_20220831T143926.SAFE | L2 from previous with CAMS
2022-09-02 10:46:19 |      30TYQ |  NOMINAL |      S2B_MSIL1C_20220902T104619_N0400_R051_T30TYQ_20220902T125405.SAFE | L2 from previous with CAMS
2022-09-05 10:56:19 |      30TYQ |  NOMINAL |      S2B_MSIL1C_20220905T105619_N0400_R094_T30TYQ_20220905T114315.SAFE | L2 from previous with CAMS
2022-09-07 10:50:41 |      30TYQ |  NOMINAL |      S2A_MSIL1C_20220907T105041_N0400_R051_T30TYQ_20220907T143122.SAFE | L2 from previous with CAMS
2022-09-10 10:56:31 |      30TYQ |  NOMINAL |      S2A_MSIL1C_20220910T105631_N0400_R094_T30TYQ_20220910T144016.SAFE | L2 from previous with CAMS
2022-09-12 10:46:29 |      30TYQ |  NOMINAL |      S2B_MSIL1C_20220912T104629_N0400_R051_T30TYQ_20220912T125224.SAFE | L2 from previous with CAMS
2023-01-06 13:26:19,889 [INFO ] Beginning workplan execution.
2023-01-06 13:26:19,889 [INFO ] Executing workplan #1/18
...
...
2023-01-07 00:42:33 [7378] PROGRESS MAJA 4.6 main:102 Nominal end of computation
2023-01-07 00:42:34,518 [INFO ] =============Start_Maja v4.6.0 finished=============

Explanations:

  • Line 4: startmaja detected the L1C products from the <repL1>
  • Line 5: startmaja did not found any pre-existing L2A. If any L2C were pre-existing, startmaja would adapt the workplan. As an example, if you already had from L2A products up to the <start date>, MAJA would use the last L2A as a restart point to continue onward in time (no need for backward)
  • Line 7: CAMS products found. This is only made if you activate the ‘–cams’ option, otherwise MAJA will use the standard ‘continental’ atmosphere model
  • Line 8: startmaja check if GIPP are available for the sensor. Here the Sentinel-2 GIPP were pre-existing. Otherwise, startmaja would download them
  • Line 10: startmaja checks for the DTM of tile 30TYQ. If not found, it would try to download it from SRTM or EUDEM repository and prepare it
  • Line 13 and onwards: startmaja show the workplan. Here, no pre-existing L2A were found, so at first MAJA will run in backward mode from the 8th to the 1st L1C product, then start from the 1st to the last in nominal mode
  • From there, startmaja launches MAJA accordingly. You may expect the backward to take about 01:20 and the following nominal processing 00:20 per tile (on a standard computer with 8 CPU)

Search