Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .nf-core.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,4 +16,4 @@ template:
name: datasync
org: nf-core
outdir: .
version: 1.0.2
version: 1.1.0dev
8 changes: 8 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,14 @@
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## v1.1.0dev

### `Added`

### `Fixed`

### `Changed`

## v1.0.2 - 2026-10-06

### `Fixed`
Expand Down
2 changes: 1 addition & 1 deletion assets/multiqc_config.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
report_comment: >
This report has been generated by the <a href="https://github.com/nf-core/datasync/releases/tag/1.0.2" target="_blank">nf-core/datasync</a> analysis pipeline. For information about how to interpret these results, please see the <a href="https://nf-co.re/datasync/1.0.2/docs/output" target="_blank">documentation</a>.
This report has been generated by the <a href="https://github.com/nf-core/datasync/tree/dev" target="_blank">nf-core/datasync</a> analysis pipeline. For information about how to interpret these results, please see the <a href="https://nf-co.re/datasync/dev/docs/output" target="_blank">documentation</a>.
report_section_order:
"nf-core-datasync-methods-description":
order: -1000
Expand Down
2 changes: 1 addition & 1 deletion nextflow.config
Original file line number Diff line number Diff line change
Expand Up @@ -293,7 +293,7 @@ manifest {
mainScript = 'main.nf'
defaultBranch = 'main'
nextflowVersion = '!>=25.10.4'
version = '1.0.2'
version = '1.1.0dev'
doi = '10.5281/zenodo.23104575'
}

Expand Down
34 changes: 17 additions & 17 deletions ro-crate-metadata.json
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,8 @@
{
"@id": "./",
"@type": "Dataset",
"creativeWorkStatus": "Stable",
"datePublished": "2026-10-06T12:16:42+00:00",
"creativeWorkStatus": "InProgress",
"datePublished": "2026-10-06T17:50:08+00:00",
"description": "<h1>\n <picture>\n <source media=\"(prefers-color-scheme: dark)\" srcset=\"docs/images/nf-core-datasync_logo_dark.png\">\n <img alt=\"nf-core/datasync\" src=\"docs/images/nf-core-datasync_logo_light.png\">\n </picture>\n</h1>\n\n[![Open in GitHub Codespaces](https://img.shields.io/badge/Open_In_GitHub_Codespaces-black?labelColor=grey&logo=github)](https://github.com/codespaces/new/nf-core/datasync)\n[![GitHub Actions CI Status](https://github.com/nf-core/datasync/actions/workflows/nf-test.yml/badge.svg)](https://github.com/nf-core/datasync/actions/workflows/nf-test.yml)\n[![GitHub Actions Linting Status](https://github.com/nf-core/datasync/actions/workflows/linting.yml/badge.svg)](https://github.com/nf-core/datasync/actions/workflows/linting.yml)[![AWS CI](https://img.shields.io/badge/CI%20tests-full%20size-FF9900?labelColor=000000&logo=Amazon%20AWS)](https://nf-co.re/datasync/results)[![Cite with Zenodo](http://img.shields.io/badge/DOI-10.5281/zenodo.23104575-1073c8?labelColor=000000)](https://doi.org/10.5281/zenodo.23104575)\n[![nf-test](https://img.shields.io/badge/unit_tests-nf--test-337ab7.svg)](https://www.nf-test.com)\n[![Nextflow](https://img.shields.io/badge/version-%E2%89%A525.10.4-green?style=flat&logo=nextflow&logoColor=white&color=%230DC09D&link=https%3A%2F%2Fnextflow.io)](https://www.nextflow.io/)\n[![nf-core template version](https://img.shields.io/badge/nf--core_template-4.1.0-green?style=flat&logo=nfcore&logoColor=white&color=%2324B064&link=https%3A%2F%2Fnf-co.re)](https://github.com/nf-core/tools/releases/tag/4.1.0)\n[![run with conda](http://img.shields.io/badge/run%20with-conda-3EB049?labelColor=000000&logo=anaconda)](https://docs.conda.io/en/latest/)\n[![run with docker](https://img.shields.io/badge/run%20with-docker-0db7ed?labelColor=000000&logo=docker)](https://www.docker.com/)\n[![run with singularity](https://img.shields.io/badge/run%20with-singularity-1d355c.svg?labelColor=000000)](https://sylabs.io/docs/)\n[![Launch on Seqera Platform](https://img.shields.io/badge/Launch%20%F0%9F%9A%80-Seqera%20Platform-%234256e7)](https://cloud.seqera.io/launch?pipeline=https://github.com/nf-core/datasync)\n[![Get help on Slack](http://img.shields.io/badge/slack-nf--core%20%23datasync-4A154B?labelColor=000000&logo=slack)](https://nfcore.slack.com/channels/datasync)[![Follow on Bluesky](https://img.shields.io/badge/bluesky-%40nf__core-1185fe?labelColor=000000&logo=bluesky)](https://bsky.app/profile/nf-co.re)[![Follow on Mastodon](https://img.shields.io/badge/mastodon-nf__core-6364ff?labelColor=FFFFFF&logo=mastodon)](https://mstdn.science/@nf_core)[![Watch on YouTube](http://img.shields.io/badge/youtube-nf--core-FF0000?labelColor=000000&logo=youtube)](https://www.youtube.com/c/nf-core)\n\n## Introduction\n\n**nf-core/datasync** is a Nextflow pipeline for copying files and directories between storage locations and documenting their integrity. For every row in an input samplesheet, the pipeline:\n\n1. validates the source against a supplied MD5 and/or SHA-256 checksum manifest using [`rclone checksum`](https://rclone.org/commands/rclone_checksum/);\n2. copies the source to the requested destination with [`rclone copy`](https://rclone.org/);\n3. compares the copied data with the source using [`rclone check`](https://rclone.org/commands/rclone_check/); and\n4. produces detailed `rclone` status files and a consolidated MultiQC report.\n\nSources and destinations may be local paths or object-storage URIs such as Amazon S3, S3-compatible storage, or Azure Blob Storage. HTTP(S) URLs are not currently supported for samplesheet `input` or `output_path` values.\n\nThe current tested use case for this pipeline is transfer between S3 buckets.\n\nPass an `rclone` configuration with `--rclone_config` whenever a source or destination URI needs credentials or provider settings. Samplesheet paths use local paths or standard URIs such as `s3://bucket/path`, not rclone's `remote:path` syntax. For non-S3 layouts, design and validate the provider-specific configuration using the upstream [rclone documentation](https://rclone.org/docs/).\n\n![nf-core/datasync metro map](docs/images/datasync_nf-metro.svg)\n\n## Quick start\n\n> [!NOTE]\n> If you are new to Nextflow and nf-core, see the [nf-core environment setup guide](https://nf-co.re/docs/get_started/environment_setup/overview). Nextflow 25.10.4 or later is required.\n\nTo explore the pipeline outputs before preparing your own data, run the bundled `test` profile with a container profile:\n\n```bash\nnextflow run nf-core/datasync \\\n -profile test,docker \\\n --outdir results\n```\n\nThe `test` profile supplies a small samplesheet and `rclone` configuration automatically. It also enables `--rclone_dry_run`, so no files are actually transferred. This makes it useful for exploring the `rclone/` output folders and `multiqc/multiqc_report.html`; remember that post-copy comparison reports describe whatever is already present at the destination because the dry run does not write transfer data.\n\nTo run the pipeline on your own data, create a samplesheet containing one transfer per row:\n\n```csv\nsample,input,output_path,checksum_md5,checksum_sha\nrun_001,/data/run_001,s3://archive/runs,/data/manifests/run_001_md5.tsv\nreference,/data/reference.fa,/data/references,,/data/manifests/reference_sha256.tsv\n```\n\nThen launch the pipeline using:\n\n```bash\nnextflow run nf-core/datasync \\\n -r <VERSION> \\\n -profile docker \\\n --input samplesheet.csv \\\n --outdir results \\\n --rclone_config /path/to/rclone.conf\n```\n\n`--rclone_config` is optional only when every source and destination is accessible without a configured rclone remote. See the [`rclone` configuration section](docs/usage.md#configuring-rclone-remotes) for the tested S3-to-S3 use case and guidance on adapting rclone configuration files for other providers. To preview copy operations without transferring data, add `--rclone_dry_run`; note that subsequent comparison reports will then describe the unchanged destination.\n\nSee the [usage documentation](docs/usage.md) for samplesheet rules, destination semantics, remote configuration, and reproducible execution. The complete generated parameter reference is available on the [nf-core pipeline page](https://nf-co.re/datasync/parameters).\n\n## Pipeline output\n\nResults are written below `--outdir`. See the [output documentation](docs/output.md) for file names and status-code interpretation.\n\n## Credits\n\nnf-core/datasync was originally written by Alexander Peltzer.\n\nWe thank the following people for their extensive assistance in the development of this pipeline:\n\n- Julian Schwab\n- Gregor Sturm\n- Antonia Saracco\n- Delfina Terradas\n- Anabella Trigila\n\n## Contributions and Support\n\nIf you would like to contribute to this pipeline, please see the [contributing guidelines](docs/CONTRIBUTING.md).\n\nFor further information or help, don't hesitate to get in touch on the [Slack `#datasync` channel](https://nfcore.slack.com/channels/datasync) (you can join with [this invite](https://nf-co.re/join/slack)).\n\n## Citations\n\nIf you use nf-core/datasync for your analysis, please cite it using the following doi: [10.5281/zenodo.23104575](https://doi.org/10.5281/zenodo.23104575)\n\nAn extensive list of references for the tools used by the pipeline can be found in the [`CITATIONS.md`](CITATIONS.md) file.\n\nYou can cite the `nf-core` publication as follows:\n\n> **The nf-core framework for community-curated bioinformatics pipelines.**\n>\n> Philip Ewels, Alexander Peltzer, Sven Fillinger, Harshil Patel, Johannes Alneberg, Andreas Wilm, Maxime Ulysse Garcia, Paolo Di Tommaso & Sven Nahnsen.\n>\n> _Nat Biotechnol._ 2020 Feb 13. doi: [10.1038/s41587-020-0439-x](https://dx.doi.org/10.1038/s41587-020-0439-x).\n",
"hasPart": [
{
Expand Down Expand Up @@ -102,7 +102,7 @@
},
"mentions": [
{
"@id": "#020e9608-3914-422a-bc82-144ce9fcf788"
"@id": "#a1898170-5367-44b1-a17f-3f2ae57384e4"
}
],
"name": "nf-core/datasync"
Expand Down Expand Up @@ -134,25 +134,25 @@
"@id": "https://orcid.org/0000-0002-6503-2180"
},
{
"@id": "#9f94a892-c2e4-4dea-9fa4-4c8d4c709492"
"@id": "#02aa1545-1c97-4abe-bbcd-e349c0511507"
},
{
"@id": "https://orcid.org/0000-0001-9584-7842"
}
],
"contributor": [
{
"@id": "#52614e4f-c318-438e-bc47-ff2a07d52931"
"@id": "#baea0247-f00c-48b6-b641-1111db4f8b11"
},
{
"@id": "#370daf34-3791-4655-8ade-5a7e148f585c"
"@id": "#17078e79-a00d-4ef1-9ad7-0c08578cd5d7"
},
{
"@id": "#cd23cb4e-3f99-4877-994f-af3ce3ea2d28"
"@id": "#e94e34fd-eefd-4a85-9982-0dd3d93fabe7"
}
],
"dateCreated": "",
"dateModified": "2026-10-06T12:16:42Z",
"dateModified": "2026-10-06T17:50:08Z",
"dct:conformsTo": "https://bioschemas.org/profiles/ComputationalWorkflow/1.0-RELEASE/",
"keywords": [
"nf-core",
Expand All @@ -172,10 +172,10 @@
},
"url": [
"https://github.com/nf-core/datasync",
"https://nf-co.re/datasync/1.0.2/"
"https://nf-co.re/datasync/dev/"
],
"version": [
"1.0.2"
"1.1.0dev"
]
},
{
Expand All @@ -191,11 +191,11 @@
"version": "!>=25.10.4"
},
{
"@id": "#020e9608-3914-422a-bc82-144ce9fcf788",
"@id": "#a1898170-5367-44b1-a17f-3f2ae57384e4",
"@type": "TestSuite",
"instance": [
{
"@id": "#a91c423b-21ed-43f8-9102-0f637d86a978"
"@id": "#71dcedb8-3d40-4621-8f83-e8fb5c6f70d3"
}
],
"mainEntity": {
Expand All @@ -204,7 +204,7 @@
"name": "Test suite for nf-core/datasync"
},
{
"@id": "#a91c423b-21ed-43f8-9102-0f637d86a978",
"@id": "#71dcedb8-3d40-4621-8f83-e8fb5c6f70d3",
"@type": "TestInstance",
"name": "GitHub Actions workflow for testing nf-core/datasync",
"resource": "repos/nf-core/datasync/actions/workflows/nf-test.yml",
Expand Down Expand Up @@ -344,25 +344,25 @@
"name": "Alexander Peltzer"
},
{
"@id": "#52614e4f-c318-438e-bc47-ff2a07d52931",
"@id": "#baea0247-f00c-48b6-b641-1111db4f8b11",
"@type": "Person",
"email": "antonia.saracco@zs.com",
"name": "Antonia Saracco"
},
{
"@id": "#370daf34-3791-4655-8ade-5a7e148f585c",
"@id": "#17078e79-a00d-4ef1-9ad7-0c08578cd5d7",
"@type": "Person",
"email": "155591053+delfiterradas@users.noreply.github.com",
"name": "Delfina Terradas"
},
{
"@id": "#cd23cb4e-3f99-4877-994f-af3ce3ea2d28",
"@id": "#e94e34fd-eefd-4a85-9982-0dd3d93fabe7",
"@type": "Person",
"email": "18577080+atrigila@users.noreply.github.com",
"name": "Anabella Trigila"
},
{
"@id": "#9f94a892-c2e4-4dea-9fa4-4c8d4c709492",
"@id": "#02aa1545-1c97-4abe-bbcd-e349c0511507",
"@type": "Person",
"name": "Julian Schwab"
},
Expand Down
4 changes: 2 additions & 2 deletions tests/default.nf.test.snap
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
"rclone": "1.74.3-DEV"
},
"Workflow": {
"nf-core/datasync": "v1.0.2"
"nf-core/datasync": "v1.1.0dev"
}
},
[
Expand Down Expand Up @@ -88,7 +88,7 @@
"benchmark_bed_checksum_MD5.missing_on_src.txt:lines,9"
]
],
"timestamp": "2026-10-06T12:21:56.462338211",
"timestamp": "2026-10-06T17:51:31.175325882",
"meta": {
"nf-test": "0.9.5",
"nextflow": "25.10.4"
Expand Down
8 changes: 4 additions & 4 deletions tests/edge.nf.test.snap
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
"rclone": "1.74.3-DEV"
},
"Workflow": {
"nf-core/datasync": "v1.0.2"
"nf-core/datasync": "v1.1.0dev"
}
},
[
Expand Down Expand Up @@ -123,7 +123,7 @@
"Illumina_annotation_sha_only_checksum_SHA256.exit_code.txt:lines,1"
]
],
"timestamp": "2026-10-06T12:22:37.58640014",
"timestamp": "2026-10-06T17:52:11.116267345",
"meta": {
"nf-test": "0.9.5",
"nextflow": "25.10.4"
Expand All @@ -142,7 +142,7 @@
"rclone": "1.74.3-DEV"
},
"Workflow": {
"nf-core/datasync": "v1.0.2"
"nf-core/datasync": "v1.1.0dev"
}
},
[
Expand Down Expand Up @@ -245,7 +245,7 @@
"Illumina_annotation_sha_only_checksum_SHA256.differ.txt:lines,15"
]
],
"timestamp": "2026-10-06T12:22:17.193039041",
"timestamp": "2026-10-06T17:51:50.915320643",
"meta": {
"nf-test": "0.9.5",
"nextflow": "25.10.4"
Expand Down
4 changes: 2 additions & 2 deletions tests/main_full.nf.test.snap
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
"rclone": "1.74.3-DEV"
},
"Workflow": {
"nf-core/datasync": "v1.0.2"
"nf-core/datasync": "v1.1.0dev"
}
},
[
Expand Down Expand Up @@ -79,7 +79,7 @@
"demultiplex_checksum_SHA256.match.txt:lines,538"
]
],
"timestamp": "2026-10-06T12:24:05.982776133",
"timestamp": "2026-10-06T17:53:59.169348145",
"meta": {
"nf-test": "0.9.5",
"nextflow": "25.10.4"
Expand Down
Loading