feature/srwgs #18
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,68 @@ | ||
| process DRAGEN { | ||
| // Dragen requires specific hardware (FPGA) to run. | ||
| // This process is therefore only executed on Illumina® BioInsight Platform Core / Illumina Connected Analytics (ICA) compute nodes with FPGA support. | ||
| // Test only contain a stub of the process, which can be executed on any compute node, but does not perform any actual processing. | ||
| tag "${meta.id}" | ||
| label 'process_fpga' | ||
|
|
||
| // Dragen 4.2.4-2 (F2 compatible container) | ||
| container '079623148045.dkr.ecr.eu-central-1.amazonaws.com/cp-prod/f1b7ad6a-11ac-4bc1-b705-b275ff2887ad:latest' | ||
|
|
||
| input: | ||
| // TODO: Move ref_tar unpacking to a separate process to avoid unpacking the reference for every sample? -> added in prs branch | ||
| tuple val(meta), path(r1_fastq), path(r2_fastq) | ||
| path fastq_list | ||
| path ref_tar | ||
| path repeat_genotype_specs | ||
|
|
||
| output: | ||
| tuple val(meta), path("*"), emit: output | ||
| tuple val(meta), path('*.csv'), emit: csv | ||
| tuple val(meta), path("${prefix}.wgs_coverage_metrics.csv"), optional: true, emit: wgs_coverage_metrics | ||
| tuple val(meta), path("${prefix}.cnv_metrics.csv"), optional: true, emit: cnv_metrics | ||
| tuple val(meta), path("${prefix}.mapping_metrics.csv"), optional: true, emit: mapping_metrics | ||
| tuple val(meta), path("${prefix}.ploidy_estimation_metrics.csv"), optional: true, emit: ploidy_estimation_metrics | ||
| tuple val(meta), path("${prefix}.gvcf_metrics.csv"), optional: true, emit: gvcf_metrics | ||
| tuple val("${task.process}"), val('dragen'), eval("dragen --version 2>&1 | sed 's/^dragen Version //'"), topic: versions, emit: versions_dragen | ||
|
|
||
| when: | ||
| task.ext.when == null || task.ext.when | ||
|
|
||
| script: | ||
| prefix = task.ext.prefix ?: "${meta.id}" | ||
| def args = task.ext.args ?: '' | ||
|
|
||
| if (repeat_genotype_specs) { | ||
| args = args + " --repeat-genotype-enable true --repeat-genotype-specs " + repeat_genotype_specs | ||
| } | ||
|
|
||
| """ | ||
| mkdir -p /scratch/reference | ||
|
There was a problem hiding this comment. I can imagine that hardcoding
Member
Author
There was a problem hiding this comment. @Jorisvansteenbrugge is 'fixing' this by creating a separate process for untarring the reference genome, see: https://github.com/UMCUGenetics/NF-Modules/blob/feature/dragen_genome_untar/modules/UMCUGenetics/untar/genome/main.nf This also makes sure that we only perform the untar once per pipeline excecution. There was a problem hiding this comment. Nice solution @Jorisvansteenbrugge ! I think it's safe to keep this as-is too. |
||
| tar -C /scratch/reference -xf ${ref_tar} | ||
|
|
||
| /opt/edico/bin/dragen --partial-reconfig HMM --ignore-version-check true | ||
|
|
||
| /opt/edico/bin/dragen --lic-instance-id-location /opt/instance-identity \\ | ||
| --ref-dir /scratch/reference/DRAGEN/9 \\ | ||
| --fastq-list ${fastq_list} \\ | ||
| --fastq-list-sample-id ${meta.id} \\ | ||
| --output-file-prefix ${prefix} \\ | ||
| --output-directory ./ \\ | ||
| --intermediate-results-dir /scratch \\ | ||
|
There was a problem hiding this comment. Same as the comment above about hardcoded
Member
Author
There was a problem hiding this comment. I think for this module it is fine to just use There was a problem hiding this comment. Makes sense. You may ignore my comment. |
||
| ${args} | ||
| """ | ||
|
|
||
| stub: | ||
| prefix = task.ext.prefix ?: "${meta.id}" | ||
| def args = task.ext.args ?: '' | ||
| """ | ||
| echo ${args} | ||
|
|
||
| touch ${prefix}.bam | ||
| touch ${prefix}.wgs_coverage_metrics.csv | ||
| touch ${prefix}.cnv_metrics.csv | ||
| touch ${prefix}.mapping_metrics.csv | ||
| touch ${prefix}.ploidy_estimation_metrics.csv | ||
| touch ${prefix}.gvcf_metrics.csv | ||
| """ | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,37 @@ | ||
| nextflow_process { | ||
|
|
||
| name "Test Process DRAGEN" | ||
| script "../main.nf" | ||
| process "DRAGEN" | ||
|
|
||
| tag "modules" | ||
| tag "modules_UMCUGenetics" | ||
| tag "dragen" | ||
|
|
||
| test("dragen - stub") { | ||
| options '-stub' | ||
| when { | ||
| process { | ||
| """ | ||
| input[0] = [ | ||
| [id:'sample1'], | ||
| [], | ||
| [] | ||
| ] | ||
| input[1] = [] | ||
| input[2] = [] | ||
| input[3] = [] | ||
| """ | ||
| } | ||
| } | ||
| then { | ||
| assertAll( | ||
| { assert process.success }, | ||
| { assert snapshot( | ||
| process.out, | ||
| process.out.findAll { key, val -> key.startsWith('versions') } | ||
| ).match() } | ||
| ) | ||
| } | ||
| } | ||
| } |
|
rernst marked this conversation as resolved.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,174 @@ | ||
| { | ||
| "dragen - stub": { | ||
| "content": [ | ||
| { | ||
| "0": [ | ||
| [ | ||
| { | ||
| "id": "sample1" | ||
| }, | ||
| [ | ||
| "sample1.bam:md5,d41d8cd98f00b204e9800998ecf8427e", | ||
| "sample1.cnv_metrics.csv:md5,d41d8cd98f00b204e9800998ecf8427e", | ||
| "sample1.gvcf_metrics.csv:md5,d41d8cd98f00b204e9800998ecf8427e", | ||
| "sample1.mapping_metrics.csv:md5,d41d8cd98f00b204e9800998ecf8427e", | ||
| "sample1.ploidy_estimation_metrics.csv:md5,d41d8cd98f00b204e9800998ecf8427e", | ||
| "sample1.wgs_coverage_metrics.csv:md5,d41d8cd98f00b204e9800998ecf8427e" | ||
| ] | ||
| ] | ||
| ], | ||
| "1": [ | ||
| [ | ||
| { | ||
| "id": "sample1" | ||
| }, | ||
| [ | ||
| "sample1.cnv_metrics.csv:md5,d41d8cd98f00b204e9800998ecf8427e", | ||
| "sample1.gvcf_metrics.csv:md5,d41d8cd98f00b204e9800998ecf8427e", | ||
| "sample1.mapping_metrics.csv:md5,d41d8cd98f00b204e9800998ecf8427e", | ||
| "sample1.ploidy_estimation_metrics.csv:md5,d41d8cd98f00b204e9800998ecf8427e", | ||
| "sample1.wgs_coverage_metrics.csv:md5,d41d8cd98f00b204e9800998ecf8427e" | ||
| ] | ||
| ] | ||
| ], | ||
| "2": [ | ||
| [ | ||
| { | ||
| "id": "sample1" | ||
| }, | ||
| "sample1.wgs_coverage_metrics.csv:md5,d41d8cd98f00b204e9800998ecf8427e" | ||
| ] | ||
| ], | ||
| "3": [ | ||
| [ | ||
| { | ||
| "id": "sample1" | ||
| }, | ||
| "sample1.cnv_metrics.csv:md5,d41d8cd98f00b204e9800998ecf8427e" | ||
| ] | ||
| ], | ||
| "4": [ | ||
| [ | ||
| { | ||
| "id": "sample1" | ||
| }, | ||
| "sample1.mapping_metrics.csv:md5,d41d8cd98f00b204e9800998ecf8427e" | ||
| ] | ||
| ], | ||
| "5": [ | ||
| [ | ||
| { | ||
| "id": "sample1" | ||
| }, | ||
| "sample1.ploidy_estimation_metrics.csv:md5,d41d8cd98f00b204e9800998ecf8427e" | ||
| ] | ||
| ], | ||
| "6": [ | ||
| [ | ||
| { | ||
| "id": "sample1" | ||
| }, | ||
| "sample1.gvcf_metrics.csv:md5,d41d8cd98f00b204e9800998ecf8427e" | ||
| ] | ||
| ], | ||
| "7": [ | ||
| [ | ||
| "DRAGEN", | ||
| "dragen", | ||
| "bash: line 1: dragen: command not found" | ||
| ] | ||
| ], | ||
| "cnv_metrics": [ | ||
| [ | ||
| { | ||
| "id": "sample1" | ||
| }, | ||
| "sample1.cnv_metrics.csv:md5,d41d8cd98f00b204e9800998ecf8427e" | ||
| ] | ||
| ], | ||
| "csv": [ | ||
| [ | ||
| { | ||
| "id": "sample1" | ||
| }, | ||
| [ | ||
| "sample1.cnv_metrics.csv:md5,d41d8cd98f00b204e9800998ecf8427e", | ||
| "sample1.gvcf_metrics.csv:md5,d41d8cd98f00b204e9800998ecf8427e", | ||
| "sample1.mapping_metrics.csv:md5,d41d8cd98f00b204e9800998ecf8427e", | ||
| "sample1.ploidy_estimation_metrics.csv:md5,d41d8cd98f00b204e9800998ecf8427e", | ||
| "sample1.wgs_coverage_metrics.csv:md5,d41d8cd98f00b204e9800998ecf8427e" | ||
| ] | ||
| ] | ||
| ], | ||
| "gvcf_metrics": [ | ||
| [ | ||
| { | ||
| "id": "sample1" | ||
| }, | ||
| "sample1.gvcf_metrics.csv:md5,d41d8cd98f00b204e9800998ecf8427e" | ||
| ] | ||
| ], | ||
| "mapping_metrics": [ | ||
| [ | ||
| { | ||
| "id": "sample1" | ||
| }, | ||
| "sample1.mapping_metrics.csv:md5,d41d8cd98f00b204e9800998ecf8427e" | ||
| ] | ||
| ], | ||
| "output": [ | ||
| [ | ||
| { | ||
| "id": "sample1" | ||
| }, | ||
| [ | ||
| "sample1.bam:md5,d41d8cd98f00b204e9800998ecf8427e", | ||
| "sample1.cnv_metrics.csv:md5,d41d8cd98f00b204e9800998ecf8427e", | ||
| "sample1.gvcf_metrics.csv:md5,d41d8cd98f00b204e9800998ecf8427e", | ||
| "sample1.mapping_metrics.csv:md5,d41d8cd98f00b204e9800998ecf8427e", | ||
| "sample1.ploidy_estimation_metrics.csv:md5,d41d8cd98f00b204e9800998ecf8427e", | ||
| "sample1.wgs_coverage_metrics.csv:md5,d41d8cd98f00b204e9800998ecf8427e" | ||
| ] | ||
| ] | ||
| ], | ||
| "ploidy_estimation_metrics": [ | ||
| [ | ||
| { | ||
| "id": "sample1" | ||
| }, | ||
| "sample1.ploidy_estimation_metrics.csv:md5,d41d8cd98f00b204e9800998ecf8427e" | ||
| ] | ||
| ], | ||
| "versions_dragen": [ | ||
| [ | ||
| "DRAGEN", | ||
| "dragen", | ||
| "bash: line 1: dragen: command not found" | ||
| ] | ||
| ], | ||
| "wgs_coverage_metrics": [ | ||
| [ | ||
| { | ||
| "id": "sample1" | ||
| }, | ||
| "sample1.wgs_coverage_metrics.csv:md5,d41d8cd98f00b204e9800998ecf8427e" | ||
| ] | ||
| ] | ||
| }, | ||
| { | ||
| "versions_dragen": [ | ||
| [ | ||
| "DRAGEN", | ||
| "dragen", | ||
| "bash: line 1: dragen: command not found" | ||
| ] | ||
| ] | ||
| } | ||
| ], | ||
| "timestamp": "2026-08-18T11:08:58.003281", | ||
| "meta": { | ||
| "nf-test": "0.9.5", | ||
| "nextflow": "25.10.2" | ||
| } | ||
| } | ||
| } | ||
|
There was a problem hiding this comment. Missing newline at the end of this file, see this StackOverflow thread on why it's important.
Member
Author
There was a problem hiding this comment. Snap files are automatically generated by nf-test. I don't think we should update/change these because they are compared using diff when running (automated) tests. |
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,37 @@ | ||
| process DRAGENUTILS_DXQC { | ||
| tag "${meta.id}" | ||
| label 'process_single' | ||
|
|
||
| container 'ghcr.io/umcugenetics/dragen-utils:1.0.0' | ||
|
|
||
| input: | ||
| tuple val(meta), path(wgs_coverage_metrics), path(cnv_metrics), path(mapping_metrics), path(ploidy_estimation_metrics), path(gvcf_metrics) | ||
|
|
||
| output: | ||
| tuple val(meta), path("${prefix}.dragen_dx_qc.csv"), emit: csv | ||
| tuple val("${task.process}"), val('dragenutils'), eval("dragen-utils --version"), topic: versions, emit: versions_dragenutils | ||
|
|
||
| when: | ||
| task.ext.when == null || task.ext.when | ||
|
|
||
| script: | ||
| prefix = task.ext.prefix ?: "${meta.id}" | ||
|
|
||
| """ | ||
| dragen-utils dx_qc \\ | ||
| ${prefix} \\ | ||
| ${wgs_coverage_metrics} \\ | ||
| ${cnv_metrics} \\ | ||
| ${mapping_metrics} \\ | ||
| ${ploidy_estimation_metrics} \\ | ||
| ${gvcf_metrics} \\ | ||
| > ${prefix}.dragen_dx_qc.csv | ||
| """ | ||
|
|
||
| stub: | ||
| prefix = task.ext.prefix ?: "${meta.id}" | ||
|
|
||
| """ | ||
| touch ${prefix}.dragen_dx_qc.csv | ||
| """ | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,40 @@ | ||
| nextflow_process { | ||
|
|
||
| name "Test Process DRAGENUTILS_DXQC" | ||
| script "../main.nf" | ||
| process "DRAGENUTILS_DXQC" | ||
|
|
||
| tag "modules" | ||
| tag "modules_UMCUGenetics" | ||
| tag "dragenutils" | ||
| tag "dragenutils/dxqc" | ||
|
|
||
| test("HG002 - srWGS QC csv files") { | ||
| when { | ||
| process { | ||
| """ | ||
| input[0] = [ | ||
| [ id:'HG002' ], | ||
| file(params.modules_umcu_testdata_base_path + "dragen_utils/qc/HG002.wgs_coverage_metrics.csv", checkIfExists: true), | ||
| file(params.modules_umcu_testdata_base_path + "dragen_utils/qc/HG002.cnv_metrics.csv", checkIfExists: true), | ||
| file(params.modules_umcu_testdata_base_path + "dragen_utils/qc/HG002.mapping_metrics.csv", checkIfExists: true), | ||
| file(params.modules_umcu_testdata_base_path + "dragen_utils/qc/HG002.ploidy_estimation_metrics.csv", checkIfExists: true), | ||
| file(params.modules_umcu_testdata_base_path + "dragen_utils/qc/HG002.gvcf_metrics.csv", checkIfExists: true), | ||
| ] | ||
| """ | ||
| } | ||
| } | ||
|
|
||
| then { | ||
| assert process.success | ||
| assertAll( | ||
| { assert snapshot( | ||
| process.out, | ||
| process.out.findAll { key, val -> key.startsWith('versions') } | ||
| ).match() } | ||
| ) | ||
| } | ||
|
|
||
| } | ||
|
|
||
| } |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
dragen --versionin the eval block fails because the dragen binary seems to be at/opt/edico/bin/dragenand is not in$PATH. I would suggest to update it to use the full path:/opt/edico/bin/dragen --version 2>&1 | sed 's/^dragen Version //'.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The problem here is that we can't run this test using the dragen container -> this proprietary software and we can't access it, furthermore we don't have FPGA nodes available within GitHub Actions. In order to make the stub test work (just testing in/output logic) we overwrite the container in https://github.com/UMCUGenetics/NF-Modules/blob/feature/srwgs/tests/config/modules.config with a simple container containing only bash.
See also nf-core example: https://github.com/nf-core/modules/blob/master/modules/nf-core/dragen/germline/tests/main.nf.test.snap
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This makes sense now. Thanks for the clarification!