Conda: OverlappingFileLockException

[ewels]

Bug report

In @nf-core we recently removed name from our conda environment.yml files. We knew that there was an issue that affect CI tests with this, but it seems that it also affects users. As discussed in nf-core/modules#6664 we are now seeing a lot people reporting this error message when running with conda:

ERROR ~ Error executing process > 'NFCORE_RNASEQ:PREPARE_GENOME:UNTAR_SALMON_INDEX'
Caused by:
  java.nio.channels.OverlappingFileLockException

Steps to reproduce the problem

@jorgee seems to have a way to reproduce this, from Slack

Launching `main.nf` [berserk_raman] DSL2 - revision: f6a195f10f
[-        ] process > greet  -
[-        ] process > greet2 -
Creating env using conda: /home/jorgee/nextflow_tests/issue-4164/conda_2/env.yaml [cache /home/jorgee/nextflow_tests/issue-4164/work/conda/env-dd83a192cf0030e98953ef467708ac6c]
ERROR ~ Error executing process > 'greet (1)'
Caused by:
  java.nio.channels.OverlappingFileLockException
 -- Check '.nextflow.log' file for details

It's typically a stochastic issue though so can be difficult to reproduce, but basically seems to happen whenever Nextflow tries to create a bunch of Conda environments at once. @jorgee says:

I suppose the error is reproduced if the pipeline is using different files with the same name but in different folders

Additional context

Some notes from Slack discussions. @pditommaso thinks it might be this code:

nextflow/modules/nextflow/src/main/groovy/nextflow/conda/CondaCache.groovy

Lines 180 to 235 in cd8c385

	String content
	String name = 'env'
	// check if it's a remote uri
	if( isYamlUriPath(condaEnv) ) {
	content = condaEnv
	}
	// check if it's a YAML file
	else if( isYamlFilePath(condaEnv) ) {
	try {
	final path = condaEnv as Path
	content = path.text
	final yaml = (Map)new Yaml().load(content)
	if( yaml.name )
	name = yaml.name
	else
	name = path.baseName
	}
	catch( NoSuchFileException e ) {
	throw new IllegalArgumentException("Conda environment file does not exist: $condaEnv")
	}
	catch( Exception e ) {
	throw new IllegalArgumentException("Error parsing Conda environment YAML file: $condaEnv -- Check the log file for details", e)
	}
	}
	else if( isTextFilePath(condaEnv) ) {
	try {
	final path = condaEnv as Path
	content = path.text
	name = path.baseName
	}
	catch( NoSuchFileException e ) {
	throw new IllegalArgumentException("Conda environment file does not exist: $condaEnv")
	}
	catch( Exception e ) {
	throw new IllegalArgumentException("Error parsing Conda environment text file: $condaEnv -- Check the log file for details", e)
	}
	}
	// it's interpreted as user provided prefix directory
	else if( condaEnv.contains('/') ) {
	final prefix = condaEnv as Path
	if( !prefix.isDirectory() )
	throw new IllegalArgumentException("Conda prefix path does not exist or is not a directory: $prefix")
	if( prefix.fileSystem != FileSystems.default )
	throw new IllegalArgumentException("Conda prefix path must be a POSIX file path: $prefix")
	return prefix
	}
	else if( condaEnv.contains('\n') ) {
	throw new IllegalArgumentException("Invalid Conda environment definition: $condaEnv")
	}
	else {
	content = condaEnv
	}
	final hash = CacheHelper.hasher(content).hash().toString()
	getCacheDir().resolve("$name-$hash")

Suggestion is that using a random id in the cache path to avoid concurrent access.

[ewels]

Trying locally I am getting the same error with both Conda and Mamba too now 🙈

$ nextflow run nf-core/rnaseq -r 3.17.0 -profile test,mamba --outdir dir

 N E X T F L O W   ~  version 24.10.0

Launching `https://github.com/nf-core/rnaseq` [clever_venter] DSL2 - revision: 00f924cf92 [3.17.0]

// [...]

Creating env using mamba: /Users/ewels/.nextflow/assets/nf-core/rnaseq/./subworkflows/local/prepare_genome/../../../modules/nf-core/gunzip/environment.yml [cache /Users/ewels/demo/mamba_test_2/work/conda/environment-cac18ad63488da9f5109ab1f794a29f6]
Creating env using mamba: /Users/ewels/.nextflow/assets/nf-core/rnaseq/./workflows/rnaseq/../../subworkflows/nf-core/fastq_qc_trim_filter_setstrandedness/../fastq_fastqc_umitools_trimgalore/../../../modules/nf-core/fastqc/environment.yml [cache /Users/ewels/demo/mamba_test_2/work/conda/environment-3ea42529974ca5a54ae5623eacfbc76a]
Creating env using mamba: /Users/ewels/.nextflow/assets/nf-core/rnaseq/./workflows/rnaseq/../../subworkflows/nf-core/fastq_qc_trim_filter_setstrandedness/../fastq_fastqc_umitools_trimgalore/../../../modules/nf-core/trimgalore/environment.yml [cache /Users/ewels/demo/mamba_test_2/work/conda/environment-7ee73282f0ad4cb3bfb895355fb91bd7]
ERROR ~ Error executing process > 'NFCORE_RNASEQ:PREPARE_GENOME:UNTAR_SALMON_INDEX'

Caused by:
  java.nio.channels.OverlappingFileLockException



 -- Check '.nextflow.log' file for details
ERROR ~ Pipeline failed. Please refer to troubleshooting docs: https://nf-co.re/docs/usage/troubleshooting

 -- Check '.nextflow.log' file for details