Unable to lock file, Tiling images failed! Error in geometric module - different from closed #213

[davedavemckay]

Hi all,
Unsure if I'm getting errors as a quirk of my setup. I am processing Sentinel-2 L1C images with force-level2 and get many failures, but not all. I'm working on a large number of images (393 .SAFE folders) and get 327 errors. e.g.

## 35 /level1/S2A_MSIL1C_20190312T114351_N0207_R123_T30UVG_20190312T151920.SAFE: dc: 54.20%. wc: 0.28%. sc: 3.97%. cc: 79.16%. AOD: 0.1874. # of targets: 62/233. Unable to lock file /level2/X0037_Y0024/20190312_LEVEL2_SEN2A_BOA.tif (timeout: 67.568300s, nx/ny: 3000/3000). Unable to lock file /level2/X0037_Y0025/20190312_LEVEL2_SEN2A_BOA.tif (timeout: 67.568300s, nx/ny: 3000/3000). error in cubing Level 2 products Tiling images failed! Error in geometric module. dc: 54.20%. wc: 0.28%. sc: 3.97%. cc: 79.16%. AOD: 0.1874. # of targets: 62/233. Unable to lock file /level2/X0037_Y0025/20190312_LEVEL2_SEN2A_BOA.tif (timeout: 67.568300s, nx/ny: 3000/3000). error in cubing Level 2 products Tiling images failed! Error in geometric module. dc: 54.20%. wc: 0.28%. sc: 3.97%. cc: 79.16%. AOD: 0.1874. # of targets: 62/233. Unable to lock file /level2/X0037_Y0025/20190312_LEVEL2_SEN2A_BOA.tif (timeout: 67.568300s, nx/ny: 3000/3000). error in cubing Level 2 products Tiling images failed! Error in geometric module. dc: 54.20%. wc: 0.28%. sc: 3.97%. cc: 79.16%. AOD: 0.1874. # of targets: 62/233. Unable to lock file /level2/X0037_Y0024/20190312_LEVEL2_SEN2A_BOA.tif (timeout: 67.568300s, nx/ny: 3000/3000). error in cubing Level 2 products Tiling images failed! Error in geometric module.

I am running, for this series, 11 concurrent Singularity containers (10 with with a queue of 36 images and a final smaller queue of 33 images). My thinking is there's a stage where my two Sentinel-2 tiles (T30UVG and T30VVH) overlap, (or S2A and S2B overlap? are these combined by force-level2?) and two runs are trying to write to the same file at the same time; one gets the file lock first and the other errors. Does this seem plausible?

• for a bit of context, the 36 images per queue setup makes sense (to me at least) as I am using NPROC=36, NTHREADS=2 on a 72-core nodes on an HPC machine.

My plan is: retry with the same parallelization scheme, but restricting to one tile at a time. Will report back!

[davedavemckay]

Update:
Running just on a single tile/sensor combination (T30VVH/S2A) improves things - I see no file lock errors. However, now I see 46/105 images failing, but can't see why. 33 images are still "QUEUED", so I'm resubmitted the job.
Assuming these 33 are part of the 46 failed images, that would leave 13 images that I need to do a bit of digging to identify.
I'm hopeful!

[thielfab]

Is there no info in the logfiles regarding the failing images? Are you using slurm on your HPC system for scheduling jobs?

[davedavemckay]

The log files just give the input file name.
Yes, it uses SLURM - unfortunately I don't have the slurm*.out files from the run with file lock error. However, it looks like there was a node failure for this attempt. RE @davidfrantz's comment below, this could point to a memory issue.

[thielfab]

You should be able to check with sacct, e.g.
sacct -j1234 --units=G --format=JobID,JobName,Partition,NodeList,Elapsed,End,State,ExitCode,AllocTRES%32,maxRSS
where 1234 is the id of your job.
If RAM was the issue, it will most likely show OUT_OF_MEMORY in the State column.

[davedavemckay]

Thanks for the tip - the state is "NODE_FAIL". The re-submission ran on a different node and gave the same error.

[davidfrantz]

Hi @davedavemckay,

my first suspect is usually memory.

for a bit of context, the 36 images per queue setup makes sense (to me at least) as I am using NPROC=36, NTHREADS=2 on a 72-core nodes on an HPC machine.

How much RAM is available on the nodes? 36 * 8GB would be required to run it safely.

Cheers,
David

[davedavemckay]

Very good point - I have 256 GB RAM to play with. Presumably that's close enough to the limit that some, but not all, are failing.

[davedavemckay]

Just to clarify, your 36 * 8, is that the NPROC=36 or the N_IMAGES=36? i.e., If I change to NPROC=18, NTHREADS=4 will that reduce the risk of hitting the memory limit?

[davidfrantz]

36 refers to NPROC, i.e. how many images are processed parallely using multiprocessing.

From what I gathered, it all points into the RAM direction. Empty logfiles can happen if the process is killed by the OS as the program did not end gracefully and cannot complete the logging. The same can happen with lockfiles when the program is killed during a write operation (program cannot remove lock, such that the next task cannot establish a lock on that file).

For further information, please see here: https://force-eo.readthedocs.io/en/latest/howto/l2-ard.html#parallel-processing

NPROC=18, NTHREADS=4 will work, I assume. Provided that you are using a single instance of force-level2 on each node (depends on how you set up SLURM). Alternatively, it might also make sense to directly invoke force-l2ps (this processes a single image) and let SLURM handle most of the parallelization (though you would need to make sure that SLURM only launches a specific number of force-l2ps instances per node - i.e. equivalently to what force-level2 uses with the NPROC parameter.

[davedavemckay]

That makes perfect sense - thanks for your help @davidfrantz and @thielfab ! No failures with 18 procs/4 threads.