diff --git a/adr/20251205-multi-revision-asset-management-strategy-pattern.md b/adr/20251205-multi-revision-asset-management-strategy-pattern.md new file mode 100644 index 0000000000..5ff3f8c4a5 --- /dev/null +++ b/adr/20251205-multi-revision-asset-management-strategy-pattern.md @@ -0,0 +1,252 @@ +# Git Multi-Revision Asset Management with Strategy Pattern + +- Authors: Jorge Ejarque +- Status: Approved +- Deciders: Jorge Ejarque, Ben Sherman, Paolo Di Tommaso +- Date: 2025-12-05 +- Tags: scm, asset-management, multi-revision + +## Summary + +Nextflow's asset management system has been refactored to support multiple revisions of the same pipeline concurrently through a bare repository approach with shared object storage, while maintaining backward compatibility with legacy direct-clone repositories using the Strategy design pattern. + +## Problem Statement + +The original asset management system (`AssetManager`) cloned each pipeline directly to `~/.nextflow/assets///.git`, creating several limitations: + +1. **No concurrent Git multi-revision support**: Only one revision of a pipeline could be checked out at a time, preventing concurrent execution of different versions +2. **Update conflicts**: Pulling updates while a pipeline was running could cause conflicts or corruption +3. **Testing limitations**: Users couldn't easily test different versions of a pipeline side-by-side + +The goal was to enable running multiple revisions of the same pipeline concurrently (e.g., production on v1.0, testing on v2.0-dev) while maintaining efficient disk usage through object sharing. + +## Goals or Decision Drivers + +- **Concurrent multi-revision execution**: Must support running different revisions of the same pipeline simultaneously +- **Efficient disk usage**: Share Git objects between revisions to minimize storage overhead +- **Backward compatibility**: Must not break existing pipelines using the legacy direct-clone approach +- **API stability**: Maintain the existing `AssetManager` API for external consumers (K8s plugin, CLI commands, etc.) +- **Minimal migration impact**: Existing repositories should continue to work without user intervention +- **JGit compatibility**: Solution must work within JGit's capabilities to avoid relying on Git client installations +- **Atomic updates**: Downloading new revisions should not interfere with running pipelines + +## Non-goals + +- **Migration of existing legacy repositories**: Legacy repos continue to work as-is; no forced migration +- **Native Git worktree support**: Due to JGit limitations, not using Git's worktree feature +- **Revision garbage collection**: No automatic cleanup of old revisions (users can manually drop) +- **Multi-hub support**: Still tied to a single repository provider per pipeline + +## Considered Options + +### Option 1: Bare Repository with Git Worktrees + +Use Git's worktree feature to create multiple working directories from a single bare repository. + +**Implementation**: +- One bare repository at `~/.nextflow/assets///.git` +- Multiple worktrees at `~/.nextflow/assets////` + +- Good, because it's the native Git solution for multiple checkouts +- Good, because worktrees are space-efficient +- Good, because Git handles all the complexity +- **Bad, because JGit doesn't support worktrees** (deal-breaker) +- Bad, because requires native Git installation + +**Decision**: Rejected due to JGit incompatibility + +### Option 2: Bare Repository + Clones per Commit + Revision Map File + +Use a bare repository for storage and create clones for each commit, tracking them in a separate file. + +**Implementation**: +- Bare repository at `~/.nextflow/assets///.nextflow/bare_repo/` +- Clones at `~/.nextflow/assets///.nextflow/commits//` +- Revision map file at `~/.nextflow/assets///.nextflow/revisions.json` mapping revision names to commit SHAs + +- Good, because it works with JGit +- Good, because bare repo reduces remote repository interactions to checkout commits +- Good, because explicit revision tracking +- Bad, because disk space as git objects replicated in clones +- Bad, because revision map file can become stale +- Bad, because requires file I/O for every revision lookup +- Bad, because potential race conditions on map file updates +- Bad, because adds complexity of maintaining external state + +**Decision**: Initially implemented but later refined + +### Option 3: Bare Repository + Shared Clones with Strategy Pattern + +Similar to Option 2 but eliminate the separate revision map file by using the bare repository itself as the source of truth. Additionally, use the Strategy pattern to maintain backward compatibility with existing legacy repositories without requiring migration. + +**Implementation**: +- Bare repository at `~/.nextflow/assets/.repos///bare/` +- Shared clones at `~/.nextflow/assets/.repos///commits//` +- Use bare repository refs to resolve revisions to commit SHAs dynamically +- JGit alternates mechanism for object sharing +- `AssetManager` as facade with unchanged public API +- `RepositoryStrategy` interface defining repository operations +- `LegacyRepositoryStrategy` for existing direct-clone behavior +- `MultiRevisionRepositoryStrategy` for new bare-repo approach +- Strategy selection based on environment variable or repository state detection + +- Good, because no external state file to maintain +- Good, because bare repository is always in sync (fetched on updates) +- Good, because simpler and more reliable +- Good, because atomic updates (Git operations are atomic) +- Good, because works entirely within JGit +- Good, because zero migration needed for existing repositories +- Good, because maintains API compatibility +- Good, because allows gradual adoption +- Good, because isolates legacy code +- Good, because makes future strategies easy to add +- Neutral, because adds abstraction layer +- Bad, because requires resolution on every access (minimal overhead) +- Bad, because increases codebase size initially + +**Decision**: Selected + +## Solution or decision outcome + +Implemented **Option 3 (Bare Repository + Shared Clones with Strategy Pattern)** for multi-revision support with backward compatibility. Multi-revision is the default for new repositories, while legacy mode is available via `NXF_SCM_LEGACY` environment variable. + +## Rationale & discussion + +### Git Multi-Revision Implementation + +The bare repository approach provides efficient multi-revision support: + +``` +~/.nextflow/assets/.repos/nextflow-io/hello/ +├── bare/ # Bare repository (shared objects) +│ ├── objects/ # All Git objects stored here +│ ├── refs/ +│ │ ├── heads/ +│ │ └── tags/ +│ └── config +│ +└── commits/ # Commit-specific clones + ├── abc123.../ # Clone for commit abc123 + │ └── .git/ + │ ├── objects/ # (uses alternates → bare/objects) + │ └── info/ + │ └── alternates # Points to bare/objects + │ + └── def456.../ # Clone for commit def456 + └── .git/ + +~/.nextflow/assets/nextflow-io/hello/ +└── .git/ # Legacy repo location (HYBRID state) +``` + +**Key mechanisms:** + +1. **Bare repository as source of truth**: The bare repo is fetched/updated from the remote, keeping refs current +2. **Dynamic resolution**: Revisions (branch/tag names) are resolved to commit SHAs using the bare repo's refs +3. **Object sharing**: Clones use Git alternates to reference the bare repo's objects, avoiding duplication +4. **Atomic operations**: Each clone is independent; downloading a new revision doesn't affect existing ones +5. **Lazy creation**: Clones are created on-demand when a specific revision is requested + +**Advantages over revision map file:** +- No external state to maintain or keep in sync +- Bare repo fetch automatically updates all refs +- Resolution is simple: `bareRepo.resolve(revision)` returns commit SHA +- No race conditions on file updates +- Simpler code with fewer failure modes + +### Strategy Pattern for Backward Compatibility + +The Strategy pattern provides clean separation and backward compatibility: + +``` +┌─────────────────────────┐ +│ AssetManager │ ← Public API (unchanged) +│ (Facade) │ +└───────────┬─────────────┘ + │ + │ delegates to + ▼ +┌─────────────────────────┐ +│ RepositoryStrategy │ ← Interface +└───────────┬─────────────┘ + △ + │ implements + ┌───────┴────────┐ + │ │ +┌───────────┐ ┌─────────────────┐ +│ Legacy │ │ MultiRevision │ ← Concrete strategies +│ Strategy │ │ Strategy │ +└───────────┘ └─────────────────┘ +``` + +**Strategy selection logic:** + +1. Check `NXF_SCM_LEGACY` environment variable → Use legacy if set +2. Check if there is only the legacy asset of the repository (`isOnlyLegacy` method) → Use legacy (preserve existing) +3. Otherwise -> Use multi-revision + + +**Backward compatibility guarantees:** + +- Existing repositories continue to work without changes +- `AssetManager` API remains identical +- CLI commands work with both strategies transparently +- Tests pass with minimal modifications +- No forced migration; users opt-in naturally when creating new repos + +### Hybrid State Handling + +The system gracefully handles hybrid states where both legacy and multi-revision repositories coexist: + +- **Detection**: In hybrid states, a multi-revision strategy is selected by default. +- **Fallback logic**: Multi-revision strategy can fall back to legacy repo for operations if needed +- **No conflicts**: Strategies are designed to coexist; operations target different directories +- **Explicit control**: Users can force a specific strategy via `setStrategyType()` or `NXF_SCM_LEGACY` environment variable + +### Migration Path + +Users naturally migrate as they pull new revisions: + +1. **Existing users**: Can continue with legacy repos (`NXF_SCM_LEGACY` state detected) +2. **New users**: Get multi-revision by default +3. **Opt-in migration**: Delete project directory to switch to multi-revision or pull with --migrate +4. **Opt-out**: Set `NXF_SCM_LEGACY=true` to force legacy mode + +### Implementation Details + +**Key classes:** + +- `RepositoryStrategy`: Interface defining repository operations +- `AbstractRepositoryStrategy`: Base class with shared helper methods +- `LegacyRepositoryStrategy`: Direct clone implementation (original behavior) +- `MultiRevisionRepositoryStrategy`: Bare repo + shared clones implementation + +**Critical methods:** + +- `download()`: Equivalent for both strategies (legacy pulls, multi-revision creates shared clone) +- `getLocalPath()`: Returns appropriate working directory based on strategy +- `getGit()`: Returns appropriate Git instance (legacy git, bare git, or commit git) + +### Performance Characteristics + +**Disk usage:** +- Legacy: ~100% per repository (full clone with all git objects) + Worktree +- Multi-revision: ~100% for bare + ~100K (.git with alternates) per revision + Worktree per revision + +**Operation speed:** +- First download: Similar (both clone from remote) +- Additional revisions: Multi-revision faster (only fetches new objects once, creates cheap clones) +- Switching revisions: Multi-revision instant (different directories), legacy requires checkout + +### Known Limitations + +- No automatic migration of legacy repositories +- Bare repository overhead even for users who only need one revision +- JGit alternates slightly more complex than worktrees +- Manual cleanup required for old revision clones + +## Links +- [GitHub Issue #2870 - Multiple revisions of the same pipeline for concurrent execution](https://github.com/nextflow-io/nextflow/issues/2870) +- [PR #6620 - Implementation of multiple revisions without revisions map](https://github.com/nextflow-io/nextflow/pull/6620) +- Related PRs implementing the multi-revision approach (linked in #6620) + diff --git a/docs/cli.md b/docs/cli.md index 01d7c77956..75c4bd07aa 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -79,6 +79,13 @@ $ nextflow run nextflow-io/hello -r v1.1 $ nextflow run nextflow-io/hello -r dev-branch $ nextflow run nextflow-io/hello -r a3f5c8e ``` +:::{versionadded} 25.12.0-edge +::: +Nextflow downloads and stores each explicitly requested Git branch, tag, or commit ID in a separate directory path, enabling you to run multiple revisions of the same pipeline simultaneously. Downloaded revisions are stored in a subdirectory of the local project: `$NXF_ASSETS/.repos///.nextflow/commits/`. + +:::{tip} +Use tags or commit IDs instead of branches for reproducible pipeline runs. Branch references change as development progresses over time. +::: (cli-params)= @@ -176,7 +183,7 @@ main script : main.nf revisions : * master (default) mybranch - v1.1 [t] +* v1.1 [t] v1.2 [t] ``` @@ -186,7 +193,7 @@ This shows: - Where it's cached locally - Which script runs by default - Available revisions (branches and tags marked with `[t]`) -- Which revision is currently checked out (marked with `*`) +- Which revisions are currently checkout out (marked with `*`) ### Pulling or updating projects diff --git a/docs/developer/diagrams/nextflow.scm.mmd b/docs/developer/diagrams/nextflow.scm.mmd index 1c52075372..e318a1bfeb 100644 --- a/docs/developer/diagrams/nextflow.scm.mmd +++ b/docs/developer/diagrams/nextflow.scm.mmd @@ -8,15 +8,53 @@ classDiagram class AssetManager { project : String - localPath : File mainScript : String - repositoryProvider : RepositoryProvider + provider : RepositoryProvider + strategy : RepositoryStrategy hub : String providerConfigs : List~ProviderConfig~ } - AssetManager --* RepositoryProvider + + class RepositoryStrategyType { + <> + LEGACY + MULTI_REVISION + } + + AssetManager --> RepositoryStrategyType + AssetManager "1" --o "1" RepositoryStrategy + AssetManager "1" --o "1" RepositoryProvider AssetManager "1" --* "*" ProviderConfig + class RepositoryStrategy { + <> + } + class AbstractRepositoryStrategy { + <> + project : String + provider : RepositoryProvider + root : File + } + class LegacyRepositoryStrategy { + localPath : File + } + class MultiRevisionRepositoryStrategy { + revision : String + bareRepo : File + commitPath : File + revisionSubdir : File + } + + RepositoryStrategy <|-- AbstractRepositoryStrategy + AbstractRepositoryStrategy <|-- LegacyRepositoryStrategy + AbstractRepositoryStrategy <|-- MultiRevisionRepositoryStrategy + + class RepositoryProvider { + <> + } + + RepositoryStrategy --> RepositoryProvider + RepositoryProvider <|-- AzureRepositoryProvider RepositoryProvider <|-- BitbucketRepositoryProvider RepositoryProvider <|-- BitbucketServerRepositoryProvider @@ -24,3 +62,4 @@ classDiagram RepositoryProvider <|-- GithubRepositoryProvider RepositoryProvider <|-- GitlabRepositoryProvider RepositoryProvider <|-- LocalRepositoryProvider + diff --git a/docs/developer/nextflow.scm.md b/docs/developer/nextflow.scm.md index ee7a4e0146..aa2df2fcb4 100644 --- a/docs/developer/nextflow.scm.md +++ b/docs/developer/nextflow.scm.md @@ -1,7 +1,7 @@ # `nextflow.scm` -The `nextflow.scm` package defines the Git provider interface and implements several built-in Git providers. +The `nextflow.scm` package defines the Git provider interface and implements several built-in Git providers. It also manages local pipeline repositories using a Strategy pattern to support different repository management approaches. ## Class Diagram @@ -12,6 +12,52 @@ The `nextflow.scm` package defines the Git provider interface and implements sev Some classes may be excluded from the above diagram for brevity. ``` -## Notes +## Architecture overview + +### Repository strategies + +The `AssetManager` uses the **Strategy pattern** to support different ways of managing local pipeline installations: + +- **`LegacyRepositoryStrategy`**: Traditional approach where each project gets a full git clone in `$NXF_HOME/{project}` directory. Only one revision can exist at a time per project. + +- **`MultiRevisionRepositoryStrategy`**: Modern approach that allows multiple revisions to coexist efficiently by: + - Maintaining a shared bare repository in `$NXF_HOME/.repos/{project}/bare/` + - Creating lightweight clones for each commit in `$NXF_HOME/.repos/{project}/commits/{commitId}/` + - Sharing git objects between revisions to minimize disk space + +### Strategy selection + +The `AssetManager` automatically selects the appropriate strategy based on: + +1. **Environment variable**: `NXF_SCM_LEGACY=true` forces legacy mode +2. **Repository status**: Detected by checking existing repository structure: + - `UNINITIALIZED`: No repository exists, use Multi-Revision (default) + - `LEGACY_ONLY`: Only legacy `.git` directory exists, use Legacy + - `BARE_ONLY`: Only bare repository exists, use Multi-Revision + - `HYBRID`: Both exist, prefer Multi-Revision + +### Repository provider + +The `RepositoryProvider` class is the base class for all Git providers. It defines how to authenticate with the provider, clone a Git repository, inspect branches and tags, etc. The provider is used by repository strategies to interact with remote Git services. + +## Key components + +### AssetManager + +Central class that manages pipeline assets. Key responsibilities include: + +- Project name resolution and validation +- Strategy selection and initialization +- Provider configuration and authentication +- Repository download and checkout operations +- Coordination between strategy and provider + +### RepositoryStrategy + +Interface defining for repository management operations: +- `download()`: Download or update a revision +- `checkout()`: Switch to a specific revision +- `drop()`: Delete local copies +- `getLocalPath()`: Get path to working directory +- `getGit()`: Access JGit repository instance -The `RepositoryProvider` class is the base class for all Git providers. It defines how to authenticate with the provider, clone a Git repository, inspect branches and tags, etc. diff --git a/docs/reference/cli.md b/docs/reference/cli.md index bf700de6bf..946892e294 100644 --- a/docs/reference/cli.md +++ b/docs/reference/cli.md @@ -263,7 +263,7 @@ The `clone` command downloads a pipeline from a Git-hosting platform into the *c : Service hub where the project is hosted. Options: `gitlab` or `bitbucket`. `-r` (`master`) -: Revision to clone - It can be a git branch, tag, or revision number. +: Revision to clone. It can be a git branch, tag, or commit SHA number. `-user` : Private repository user name. @@ -314,6 +314,11 @@ The `config` command is used for printing the project's configuration i.e. the ` `-properties` : Print config using Java properties notation. +`-r, -revision` +: :::{versionadded} 25.12.0-edge + ::: +: Project revision. Can be a git branch, tag, or commit SHA number. + `-a, -show-profiles` : Show all configuration profiles. @@ -442,6 +447,11 @@ The `drop` command is used to remove the projects which have been downloaded int `-h, -help` : Print the command usage. +`-r, -revision` +: :::{versionadded} 25.12.0-edge +::: +: Project revision to drop. Can be a git branch, tag, or commit SHA number. + **Examples** Drop the `nextflow-io/hello` project. @@ -612,7 +622,7 @@ $ nextflow info nextflow-io/hello * master (default) mybranch testing - v1.1 [t] + * v1.1 [t] v1.2 [t] ``` @@ -1155,10 +1165,13 @@ The `pull` command downloads a pipeline from a Git-hosting platform into the glo **Options** -`-all` +`-a, -all` : Update all downloaded projects. `-d, -deep` +: :::{deprecated} 25.12.0-edge. + Ignored for new multi-revision asset management strategy. Still used in legacy assets. + ::: : Create a shallow clone of the specified depth. `-h, -help` @@ -1167,8 +1180,13 @@ The `pull` command downloads a pipeline from a Git-hosting platform into the glo `-hub` (`github`) : Service hub where the project is hosted. Options: `gitlab` or `bitbucket` +`-migrate` +:::{versionadded} 25.12.0-edge + ::: +: Update the project asset to new multi-revision strategy. + `-r, -revision` -: Revision of the project to run (either a git branch, tag or commit hash). +: Project revision to run. Can be a git branch, tag, or commit SHA number. : When passing a git tag or branch, the `workflow.revision` and `workflow.commitId` fields are populated. When passing only the commit hash, `workflow.revision` is not defined. `-user` @@ -1236,6 +1254,9 @@ The `run` command is used to execute a local pipeline script or remote pipeline : Enable/disable processes caching. `-d, -deep` +: :::{deprecated} 25.12.0-edge + Ignored for new multi-revision asset management strategy. Still used in legacy assets. + ::: : Create a shallow clone of the specified depth. `-disable-jobs-cancellation` @@ -1319,7 +1340,7 @@ The `run` command is used to execute a local pipeline script or remote pipeline : Execute the script using the cached results, useful to continue executions that was stopped by an error. `-r, -revision` -: Revision of the project to run (either a git branch, tag or commit hash). +: Project revision to run. Can be a git branch, tag, or commit SHA number. : When passing a git tag or branch, the `workflow.revision` and `workflow.commitId` fields are populated. When passing only the commit hash, `workflow.revision` is not defined. `-stub-run, -stub` @@ -1565,6 +1586,11 @@ The `view` command is used to inspect the pipelines that are already stored in t `-q` : Hide header line. +`-r, -revision` +: :::{versionadded} 25.12.0-edge + ::: +: Project revision. Can be a git branch, tag, or commit SHA number. + **Examples** Viewing the contents of a downloaded pipeline. diff --git a/modules/nextflow/src/main/groovy/nextflow/cli/CmdClone.groovy b/modules/nextflow/src/main/groovy/nextflow/cli/CmdClone.groovy index 9c582aaaf6..511f4f7ce1 100644 --- a/modules/nextflow/src/main/groovy/nextflow/cli/CmdClone.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/cli/CmdClone.groovy @@ -37,7 +37,7 @@ class CmdClone extends CmdBase implements HubOptions { @Parameter(required=true, description = 'name of the project to clone') List args - @Parameter(names='-r', description = 'Revision to clone - It can be a git branch, tag or revision number') + @Parameter(names='-r', description = 'Revision of the project to clone (either a git branch, tag or commit SHA number)') String revision @Parameter(names=['-d','-deep'], description = 'Create a shallow clone of the specified depth') @@ -68,9 +68,9 @@ class CmdClone extends CmdBase implements HubOptions { } manager.checkValidRemoteRepo() - print "Cloning ${manager.project}${revision ? ':'+revision:''} ..." + print "Cloning ${manager.getProjectWithRevision()} ..." manager.clone(target, revision, deep) print "\r" - println "${manager.project} cloned to: $target" + println "${manager.getProjectWithRevision()} cloned to: $target" } } diff --git a/modules/nextflow/src/main/groovy/nextflow/cli/CmdConfig.groovy b/modules/nextflow/src/main/groovy/nextflow/cli/CmdConfig.groovy index c4fe0f681b..9d4da3b908 100644 --- a/modules/nextflow/src/main/groovy/nextflow/cli/CmdConfig.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/cli/CmdConfig.groovy @@ -48,6 +48,9 @@ class CmdConfig extends CmdBase { @Parameter(description = 'project name') List args = [] + @Parameter(names=['-r','-revision'], description = 'Revision of the project (either a git branch, tag or commit SHA number)') + String revision + @Parameter(names=['-a','-show-profiles'], description = 'Show all configuration profiles') boolean showAllProfiles @@ -230,7 +233,11 @@ class CmdConfig extends CmdBase { return file.parent ?: Paths.get('/') } - final manager = new AssetManager(path) + final manager = new AssetManager(path, revision) + if( revision && manager.isUsingLegacyStrategy() ){ + log.warn("The local asset for ${path} does not support multi-revision - 'revision' option is ignored\n" + + "Consider updating the project using 'nextflow pull ${path} -r $revision -migrate'") + } manager.isLocal() ? manager.localPath.toPath() : manager.configFile?.parent } diff --git a/modules/nextflow/src/main/groovy/nextflow/cli/CmdDrop.groovy b/modules/nextflow/src/main/groovy/nextflow/cli/CmdDrop.groovy index 9d67190a54..e6de1d192c 100644 --- a/modules/nextflow/src/main/groovy/nextflow/cli/CmdDrop.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/cli/CmdDrop.groovy @@ -20,7 +20,6 @@ import com.beust.jcommander.Parameter import com.beust.jcommander.Parameters import groovy.transform.CompileStatic import groovy.util.logging.Slf4j -import nextflow.exception.AbortOperationException import nextflow.plugin.Plugins import nextflow.scm.AssetManager @@ -39,6 +38,9 @@ class CmdDrop extends CmdBase { @Parameter(required=true, description = 'name of the project to drop') List args + @Parameter(names=['-r','-revision'], description = 'Revision of the project to drop (either a git branch, tag or commit SHA number)') + String revision + @Parameter(names='-f', description = 'Delete the repository without taking care of local changes') boolean force @@ -48,18 +50,6 @@ class CmdDrop extends CmdBase { @Override void run() { Plugins.init() - def manager = new AssetManager(args[0]) - if( !manager.localPath.exists() ) { - throw new AbortOperationException("No match found for: ${args[0]}") - } - - if( this.force || manager.isClean() ) { - manager.close() - if( !manager.localPath.deleteDir() ) - throw new AbortOperationException("Unable to delete project `${manager.project}` -- Check access permissions for path: ${manager.localPath}") - return - } - - throw new AbortOperationException("Local project repository contains uncommitted changes -- won't drop it") + new AssetManager(args[0]).drop(revision, force) } } diff --git a/modules/nextflow/src/main/groovy/nextflow/cli/CmdInfo.groovy b/modules/nextflow/src/main/groovy/nextflow/cli/CmdInfo.groovy index c8c1c31bc0..e2beffff10 100644 --- a/modules/nextflow/src/main/groovy/nextflow/cli/CmdInfo.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/cli/CmdInfo.groovy @@ -76,8 +76,9 @@ class CmdInfo extends CmdBase { Plugins.init() final manager = new AssetManager(args[0]) - if( !manager.isLocal() ) + if( manager.isNotInitialized() ) { throw new AbortOperationException("Unknown project `${args[0]}`") + } if( !format || format == 'text' ) { printText(manager,level) @@ -101,7 +102,7 @@ class CmdInfo extends CmdBase { out.println " project name: ${manager.project}" out.println " repository : ${manager.repositoryUrl}" - out.println " local path : ${manager.localPath}" + out.println " local path : ${manager.projectPath}" out.println " main script : ${manager.mainScriptName}" if( manager.homePage && manager.homePage != manager.repositoryUrl ) out.println " home page : ${manager.homePage}" @@ -138,7 +139,7 @@ class CmdInfo extends CmdBase { def result = [:] result.projectName = manager.project result.repository = manager.repositoryUrl - result.localPath = manager.localPath?.toString() + result.localPath = manager.projectPath?.toString() result.manifest = manager.manifest.toMap() result.revisions = manager.getBranchesAndTags(checkForUpdates) return result diff --git a/modules/nextflow/src/main/groovy/nextflow/cli/CmdPull.groovy b/modules/nextflow/src/main/groovy/nextflow/cli/CmdPull.groovy index 0c99430964..c662df99ec 100644 --- a/modules/nextflow/src/main/groovy/nextflow/cli/CmdPull.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/cli/CmdPull.groovy @@ -38,15 +38,18 @@ class CmdPull extends CmdBase implements HubOptions { @Parameter(description = 'project name or repository url to pull', arity = 1) List args - @Parameter(names='-all', description = 'Update all downloaded projects', arity = 0) + @Parameter(names=['-a','-all'], description = 'Update all downloaded projects', arity = 0) boolean all - @Parameter(names=['-r','-revision'], description = 'Revision of the project to run (either a git branch, tag or commit SHA number)') + @Parameter(names=['-r','-revision'], description = 'Revision of the project to pull (either a git branch, tag or commit SHA number)') String revision @Parameter(names=['-d','-deep'], description = 'Create a shallow clone of the specified depth') Integer deep + @Parameter(names=['-m','-migrate'], description = 'Migrate projects to multi-revision strategy', arity = 0) + boolean migrate + @Override final String getName() { NAME } @@ -59,6 +62,12 @@ class CmdPull extends CmdBase implements HubOptions { if( !all && !args ) throw new AbortOperationException('Missing argument') + if( all && args ) + throw new AbortOperationException('Option `all` requires no arguments') + + if( all && revision ) + throw new AbortOperationException('Option `all` is not compatible with `revision`') + def list = all ? AssetManager.list() : args.toList() if( !list ) { log.info "(nothing to do)" @@ -71,20 +80,42 @@ class CmdPull extends CmdBase implements HubOptions { // init plugin system Plugins.init() - - list.each { - log.info "Checking $it ..." - def manager = new AssetManager(it, this) - - def result = manager.download(revision,deep) - manager.updateModules() - - def scriptFile = manager.getScriptFile() - String message = !result ? " done" : " $result" - message += " - revision: ${scriptFile.revisionInfo}" - log.info message + + for( String proj : list ) { + if( all ) { + def branches = new AssetManager(proj).getBranchesAndTags(false).pulled as List + branches.each { rev -> pullProjectRevision(proj, rev) } + } else { + pullProjectRevision(proj, revision) + } + } + } + + private pullProjectRevision(String project, String revision) { + final manager = new AssetManager(project, this) + + if( manager.isUsingLegacyStrategy() ) { + if( migrate ) { + log.info "Migrating ${project} revision ${revision} to multi-revision strategy" + manager.setStrategyType(AssetManager.RepositoryStrategyType.MULTI_REVISION) + } else { + log.warn "The local asset for ${project} does not support multi-revision - Pulling with legacy strategy\n" + + "Consider updating the project ${project} using '-migrate' option" + } } + if( revision ) + manager.setRevision(revision) + + log.info "Checking ${manager.getProjectWithRevision()} ..." + + def result = manager.download(revision, deep) + manager.updateModules() + + def scriptFile = manager.getScriptFile() + String message = !result ? " done" : " $result" + message += " - revision: ${scriptFile.revisionInfo}" + log.info message } } diff --git a/modules/nextflow/src/main/groovy/nextflow/cli/CmdRun.groovy b/modules/nextflow/src/main/groovy/nextflow/cli/CmdRun.groovy index 2c625e23c6..309ef27d3f 100644 --- a/modules/nextflow/src/main/groovy/nextflow/cli/CmdRun.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/cli/CmdRun.groovy @@ -636,7 +636,9 @@ class CmdRun extends CmdBase implements HubOptions { * try to look for a pipeline in the repository */ def manager = new AssetManager(pipelineName, this) - def repo = manager.getProject() + if( revision ) + manager.setRevision(revision) + def repo = manager.getProjectWithRevision() boolean checkForUpdate = true if( !manager.isRunnable() || latest ) { @@ -648,7 +650,12 @@ class CmdRun extends CmdBase implements HubOptions { log.info " $result" checkForUpdate = false } - // checkout requested revision + // Warn if using legacy + if( manager.isUsingLegacyStrategy() ){ + log.warn1 "This Nextflow version supports a new Multi-revision strategy for managing the SCM repositories, " + + "but '${repo}' is single-revision legacy strategy - Please consider to update the repository with the 'nextflow pull -migrate' command." + } + // post download operations try { manager.checkout(revision) manager.updateModules() diff --git a/modules/nextflow/src/main/groovy/nextflow/cli/CmdView.groovy b/modules/nextflow/src/main/groovy/nextflow/cli/CmdView.groovy index dcdcfcae67..ef0b787339 100644 --- a/modules/nextflow/src/main/groovy/nextflow/cli/CmdView.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/cli/CmdView.groovy @@ -42,6 +42,9 @@ class CmdView extends CmdBase { @Parameter(description = 'project name', required = true) List args = [] + @Parameter(names=['-r','-revision'], description = 'Revision of the project (either a git branch, tag or commit SHA number)') + String revision + @Parameter(names = '-q', description = 'Hide header line', arity = 0) boolean quiet @@ -51,10 +54,13 @@ class CmdView extends CmdBase { @Override void run() { Plugins.init() - def manager = new AssetManager(args[0]) + final manager = new AssetManager(args[0], revision) if( !manager.isLocal() ) - throw new AbortOperationException("Unknown project name `${args[0]}`") - + throw new AbortOperationException("Unknown project `${manager.getProjectWithRevision()}`") + if( revision && manager.isUsingLegacyStrategy()){ + log.warn("The local asset ${args[0]} does not support multi-revision - 'revision' option is ignored\n" + + "Consider updating the asset using 'nextflow pull ${args[0]} -r $revision -migrate'") + } if( all ) { if( !quiet ) println "== content of path: ${manager.localPath}" diff --git a/modules/nextflow/src/main/groovy/nextflow/scm/AbstractRepositoryStrategy.groovy b/modules/nextflow/src/main/groovy/nextflow/scm/AbstractRepositoryStrategy.groovy new file mode 100644 index 0000000000..906aa44a38 --- /dev/null +++ b/modules/nextflow/src/main/groovy/nextflow/scm/AbstractRepositoryStrategy.groovy @@ -0,0 +1,131 @@ +/* + * Copyright 2013-2025, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.scm + +import groovy.transform.CompileStatic +import groovy.util.logging.Slf4j +import nextflow.Const +import nextflow.config.Manifest +import org.eclipse.jgit.lib.Constants +import org.eclipse.jgit.lib.ObjectId +import org.eclipse.jgit.lib.Ref +import org.eclipse.jgit.lib.Repository + +/** + * Abstract base class providing common repository operations shared by all strategies. + * Implements the Template Method pattern where concrete strategies override specific + * behaviors while sharing common helper methods. + * + * @author Jorge Ejarque + */ +@Slf4j +@CompileStatic +abstract class AbstractRepositoryStrategy implements RepositoryStrategy { + + protected static final String REMOTE_REFS_ROOT = "refs/remotes/origin/" + protected static final String REMOTE_DEFAULT_HEAD = REMOTE_REFS_ROOT + "HEAD" + + /** + * Context providing access to shared resources and configuration + */ + protected final File root = AssetManager.root + protected String project + protected RepositoryProvider provider + + AbstractRepositoryStrategy(String project) { + this.project = project + } + + AbstractRepositoryStrategy(String project, RepositoryProvider provider) { + this.project = project + this.provider = provider + } + + void setProvider(RepositoryProvider provider) { + this.provider = provider + } + + void setProject(String project) { + this.project = project + } + + String getCurrentRevision() { + Ref head = findHeadRef() + if( !head ) + return '(unknown)' + + if( head.isSymbolic() ) + return Repository.shortenRefName(head.getTarget().getName()) + + if( !head.getObjectId() ) + return '(unknown)' + + // try to resolve the current object id to a tag name + def name = resolveTagNameByObjectId(head.objectId) + return name ? Repository.shortenRefName(name) : head.objectId.name() + } + + AssetManager.RevisionInfo getCurrentRevisionAndName() { + Ref head = findHeadRef() + if( !head ) + return null + + if( head.isSymbolic() ) { + return new AssetManager.RevisionInfo( + head.objectId.name(), + Repository.shortenRefName(head.getTarget().getName()), + AssetManager.RevisionInfo.Type.BRANCH + ) + } + + if( !head.getObjectId() ) + return null + + final name = resolveTagNameByObjectId(head.objectId) + if( name ) { + return new AssetManager.RevisionInfo(head.objectId.name(), Repository.shortenRefName(name), AssetManager.RevisionInfo.Type.TAG) + } else { + return new AssetManager.RevisionInfo(head.objectId.name(), null, AssetManager.RevisionInfo.Type.COMMIT) + } + } + + /** + * Find the HEAD reference + */ + protected Ref findHeadRef() { + getGit()?.getRepository()?.findRef(Constants.HEAD) + } + + /** + * Try to resolve an object id to a tag name + */ + protected String resolveTagNameByObjectId(ObjectId objectId) { + Collection tags = getGit()?.getRepository()?.getRefDatabase()?.getRefsByPrefix(Constants.R_TAGS) + return tags?.find { it.objectId == objectId || it.peeledObjectId == objectId }?.name + } + + /** + * Get the default branch name + */ + protected String getDefaultBranch(Manifest manifest) { + return manifest?.getDefaultBranch() + ?: getRemoteDefaultBranch() + ?: Const.DEFAULT_BRANCH + } + + +} diff --git a/modules/nextflow/src/main/groovy/nextflow/scm/AssetManager.groovy b/modules/nextflow/src/main/groovy/nextflow/scm/AssetManager.groovy index b35f1a06d2..2bddddd5fa 100644 --- a/modules/nextflow/src/main/groovy/nextflow/scm/AssetManager.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/scm/AssetManager.groovy @@ -17,6 +17,7 @@ package nextflow.scm import static nextflow.Const.* +import static GitReferenceHelper.* import java.nio.file.Path @@ -33,21 +34,24 @@ import nextflow.config.ConfigParserFactory import nextflow.exception.AbortOperationException import nextflow.exception.AmbiguousPipelineNameException import nextflow.script.ScriptFile +import nextflow.SysEnv import nextflow.util.IniFile -import org.eclipse.jgit.api.CreateBranchCommand import org.eclipse.jgit.api.Git -import org.eclipse.jgit.api.ListBranchCommand -import org.eclipse.jgit.api.MergeResult import org.eclipse.jgit.api.errors.RefNotFoundException -import org.eclipse.jgit.errors.RepositoryNotFoundException -import org.eclipse.jgit.lib.Constants -import org.eclipse.jgit.lib.ObjectId import org.eclipse.jgit.lib.Ref import org.eclipse.jgit.lib.Repository -import org.eclipse.jgit.lib.SubmoduleConfig.FetchRecurseSubmodulesMode import org.eclipse.jgit.merge.MergeStrategy /** - * Handles operation on remote and local installed pipelines + * Handles operation on remote and local installed pipelines. + * Uses the strategy pattern to support different ways to manage local installations. + * Current available {@link RepositoryStrategy}: + * - {@link LegacyRepositoryStrategy}: This is the traditional approach where each project gets a full git clone. + * - {@Link MultiRevisionRepositoryStrategy}: This approach allows multiple revisions to coexist efficiently by sharing objects + * through a bare repository and creating lightweight clones for each commit. + * + * A {@link AssetManager.RepositoryStatus} is defined according to the status of the project folder (`localRootPath`). + * It is used to automatically select the {@link RepositoryStrategy}. The {@link LegacyRepositoryStrategy} will be selected for LEGACY_ONLY status, + * and the @Link MultiRevisionRepositoryStrategy} for other statuses (UNINNITIALIZED, BARE_ONLY and HYBRID) * * @author Paolo Di Tommaso */ @@ -55,8 +59,6 @@ import org.eclipse.jgit.merge.MergeStrategy @Slf4j @CompileStatic class AssetManager { - private static final String REMOTE_REFS_ROOT = "refs/remotes/origin/" - private static final String REMOTE_DEFAULT_HEAD = REMOTE_REFS_ROOT + "HEAD" /** * The folder all pipelines scripts are installed @@ -71,11 +73,6 @@ class AssetManager { */ private String project - /** - * Directory where the pipeline is cloned (i.e. downloaded) - */ - private File localPath - private Git _git private String mainScript @@ -86,6 +83,11 @@ class AssetManager { private List providerConfigs + /** + * The repository strategy (legacy or multi-revision) + */ + private RepositoryStrategy strategy + /** * Create a new asset manager object with default parameters */ @@ -112,6 +114,14 @@ class AssetManager { build(pipelineName, config) } + AssetManager( String pipelineName, String revision, HubOptions cliOpts = null ) { + assert pipelineName + // build the object + def config = ProviderConfig.getDefault() + // build the object + build(pipelineName, config, cliOpts, revision) + } + /** * Build the asset manager internal data structure * @@ -121,27 +131,185 @@ class AssetManager { * @return The {@link AssetManager} object itself */ @PackageScope - AssetManager build( String pipelineName, Map config = null, HubOptions cliOpts = null ) { + AssetManager build( String pipelineName, Map config = null, HubOptions cliOpts = null, String revision = null ) { this.providerConfigs = ProviderConfig.createFromMap(config) this.project = resolveName(pipelineName) - this.localPath = checkProjectDir(project) + + if( !isValidProjectName(this.project) ) { + throw new IllegalArgumentException("Not a valid project name: ${this.project}") + } + // Initialize strategy based on environment and repository state + initStrategy(revision) this.hub = checkHubProvider(cliOpts) this.provider = createHubProvider(hub) + + if( revision ){ + setRevision(revision) + } + + strategy.setProvider(this.provider) + setupCredentials(cliOpts) + validateProjectDir() return this } + /** + * Update the repository strategy based on environment and current state, or create a new one if not exists + */ + private void updateStrategy(RepositoryStrategyType type = null) { + def revision = getRevision() + if( strategy ) + strategy.close() + if( !type ) + type = selectStrategyType() + if( type == RepositoryStrategyType.LEGACY ) + strategy = new LegacyRepositoryStrategy(project) + else if( type == RepositoryStrategyType.MULTI_REVISION ) + strategy = new MultiRevisionRepositoryStrategy(project, revision) + else + throw new AbortOperationException("Not supported strategy $type") + if( revision ) + setRevision(revision) + strategy.setProvider(provider) + } + + private void initStrategy(String revision = null) { + def type = selectStrategyType() + if( type == RepositoryStrategyType.LEGACY ) + strategy = new LegacyRepositoryStrategy(project) + else if( type == RepositoryStrategyType.MULTI_REVISION ) + strategy = new MultiRevisionRepositoryStrategy(project, revision) + else + throw new AbortOperationException("Not supported strategy $type") + if( revision ) + setRevision(revision) + strategy.setProvider(provider) + } + + private RepositoryStrategyType selectStrategyType() { + // Check environment variable for legacy mode + if( SysEnv.get('NXF_SCM_LEGACY') as boolean ) { + log.warn "Forcing to use legacy repository strategy (NXF_SCM_LEGACY is set to true)" + return RepositoryStrategyType.LEGACY + } + + if (isOnlyLegacy()){ + log.debug "Legacy repository detected, selecting legacy strategy" + return RepositoryStrategyType.LEGACY + } else { + log.debug "Selecting multi-revision strategy" + return RepositoryStrategyType.MULTI_REVISION + } + } + + /** + * Set the repository strategy type explicitly + * + * @param useLegacy If true, use legacy strategy; otherwise use multi-revision + */ + void setStrategyType(RepositoryStrategyType type) { + if( (type == RepositoryStrategyType.LEGACY && isUsingLegacyStrategy()) || + (type == RepositoryStrategyType.MULTI_REVISION && isUsingMultiRevisionStrategy()) ) { + //nothing to do + return + } + if( type == RepositoryStrategyType.LEGACY && isUsingMultiRevisionStrategy() ) { + log.warn1 "Switching to legacy SCM repository management" + updateStrategy(RepositoryStrategyType.LEGACY) + } else if( type == RepositoryStrategyType.MULTI_REVISION && SysEnv.get('NXF_SCM_LEGACY') ) { + log.warn1 "Received a request to change to multi-revision SCM repository management and NXF_SCM_LEGACY is defined. Keeping legacy strategy" + updateStrategy(RepositoryStrategyType.LEGACY) + } else { + log.debug "Switching to multi-revision repository strategy" + updateStrategy(RepositoryStrategyType.MULTI_REVISION) + } + } + + /** + * Check if using legacy strategy + */ + boolean isUsingLegacyStrategy() { + return strategy instanceof LegacyRepositoryStrategy + } + + /** + * Check if using multi-revision strategy + */ + boolean isUsingMultiRevisionStrategy() { + return strategy instanceof MultiRevisionRepositoryStrategy + } + + /** + * Check if a multi-revision or legacy local asset exist. + * @return 'true' if none of them exist, otherwise 'false'. + */ + boolean isNotInitialized() { + final hasMultiRevision = MultiRevisionRepositoryStrategy.checkProject(root, project) + final hasLegacy = LegacyRepositoryStrategy.checkProject(root, project) + + return !hasMultiRevision && !hasLegacy + } + + /** + * Check if repository has only a local legacy asset + * @return 'true' if legacy asset exists and multi-revision doesn't exist, otherwise 'false'. + */ + boolean isOnlyLegacy(){ + final hasMultiRevision = MultiRevisionRepositoryStrategy.checkProject(root, project) + final hasLegacy = LegacyRepositoryStrategy.checkProject(root, project) + + return hasLegacy && !hasMultiRevision + } + + + /** + * Set the revision for multi-revision strategy + */ + AssetManager setRevision(String revision) { + if( provider ) + provider.setRevision(revision) + if( revision && strategy instanceof MultiRevisionRepositoryStrategy ) { + ((MultiRevisionRepositoryStrategy) strategy).setRevision(revision) + } + return this + } + + /** + * Get the current revision (multi-revision strategy only) + */ + private String getRevision() { + if( provider ) + return provider.getRevision() + if( strategy && strategy instanceof MultiRevisionRepositoryStrategy ) { + return ((MultiRevisionRepositoryStrategy) strategy).getRevision() + } + return null + } + + /** + * Get the project name with revision appended (for multi-revision strategy) + */ + String getProjectWithRevision() { + def rev = getRevision() + return project + (rev ? ':' + rev : '') + } + @PackageScope File getLocalGitConfig() { - localPath ? new File(localPath,'.git/config') : null + return strategy?.getLocalGitConfig() ?: null } - @PackageScope AssetManager setProject(String name) { + @PackageScope + AssetManager setProject(String name) { this.project = name + if( !strategy ) + initStrategy(getRevision()) + strategy.setProject(name) return this } @@ -170,23 +338,6 @@ class AssetManager { projectName =~~ /.+\/.+/ } - /** - * Verify the project name matcher the expected pattern. - * and return the directory where the project is stored locally - * - * @param projectName A project name matching the pattern {@code owner/project} - * @return The project dir {@link File} - */ - @PackageScope - File checkProjectDir(String projectName) { - - if( !isValidProjectName(projectName)) { - throw new IllegalArgumentException("Not a valid project name: $projectName") - } - - new File(root, project) - } - /** * Verifies that the project hub provider eventually specified by the user using the {@code -hub} command * line option or implicitly by entering a repository URL, matches with clone URL of a project already cloned (downloaded). @@ -361,7 +512,9 @@ class AssetManager { } AssetManager setLocalPath(File path) { - this.localPath = path + if (!strategy) + initStrategy(getRevision()) + strategy.setLocalPath(path) return this } @@ -375,17 +528,17 @@ class AssetManager { return this } - @Memoized String getGitRepositoryUrl() { + return strategy?.getGitRepositoryUrl() ?: provider.getCloneUrl() + } - if( localPath.exists() ) { - return localPath.toURI().toString() - } - - provider.getCloneUrl() + File getLocalPath() { + return strategy?.getLocalPath() } - File getLocalPath() { localPath } + File getProjectPath() { + return strategy?.getProjectPath() + } ScriptFile getScriptFile(String scriptName=null) { @@ -427,20 +580,16 @@ class AssetManager { // if specified in manifest, that takes priority // otherwise look for a symbolic ref (refs/remotes/origin/HEAD) return getManifest().getDefaultBranch() - ?: getRemoteBranch() + ?: strategy?.getRemoteDefaultBranch() ?: DEFAULT_BRANCH } - protected String getRemoteBranch() { - Ref remoteHead = git.getRepository().findRef(REMOTE_DEFAULT_HEAD) - return remoteHead?.getTarget()?.getName()?.substring(REMOTE_REFS_ROOT.length()) - } - @Memoized Manifest getManifest() { getManifest0() } + protected Manifest getManifest0() { String text = null ConfigObject result = null @@ -496,7 +645,7 @@ class AssetManager { } boolean isLocal() { - localPath.exists() + return localPath && localPath.exists() } /** @@ -504,7 +653,7 @@ class AssetManager { * file (i.e. main.nf) or the nextflow manifest file (i.e. nextflow.config) */ boolean isRunnable() { - localPath.exists() && ( new File(localPath,DEFAULT_MAIN_FILE_NAME).exists() || new File(localPath,MANIFEST_FILE_NAME).exists() ) + isLocal() && ( new File(localPath,DEFAULT_MAIN_FILE_NAME).exists() || new File(localPath,MANIFEST_FILE_NAME).exists() ) } /** @@ -512,12 +661,7 @@ class AssetManager { * and the current HEAD, false if differences do exist */ boolean isClean() { - try { - git.status().call().isClean() - } - catch( RepositoryNotFoundException e ) { - return true - } + return strategy?.isClean() ?: true } /** @@ -528,6 +672,9 @@ class AssetManager { _git.close() _git = null } + if( strategy ) { + strategy.close() + } } /** @@ -536,17 +683,30 @@ class AssetManager { static List list() { log.debug "Listing projects in folder: $root" - def result = new LinkedList() if( !root.exists() ) - return result + return [] - root.eachDir { File org -> - org.eachDir { File it -> - result << "${org.getName()}/${it.getName()}".toString() + def result = new HashSet() + appendProjectsFromDir(root, result) + return result.toList().sort() + } + + /** + * Append found projects to results considering the new multi-revision projects in '.repos' subdirectory + * @param dir + * @param result + */ + private static void appendProjectsFromDir(File dir, HashSet result) { + dir.eachDir { File org -> + if( org.getName() == MultiRevisionRepositoryStrategy.REPOS_SUBDIR ) { + appendProjectsFromDir(org, result) + } else { + org.eachDir { File it -> + log.debug("Adding ${org.getName()}/${it.getName()}") + result << "${org.getName()}/${it.getName()}".toString() + } } } - - return result } static protected def find( String name ) { @@ -567,6 +727,10 @@ class AssetManager { protected Git getGit() { + if( strategy ) { + return strategy.getGit() + } + // Fallback to legacy behavior if strategy not initialized if( !_git ) { _git = Git.open(localPath) } @@ -577,112 +741,19 @@ class AssetManager { * Download a pipeline from a remote Github repository * * @param revision The revision to download - * @result A message representing the operation result + * @param deep Optional depth for shallow clones + * @return A message representing the operation result */ - String download(String revision=null, Integer deep=null) { + String download(String revision = null, Integer deep = null) { assert project - - /* - * if the pipeline already exists locally pull it from the remote repo - */ + if( !strategy ) { + throw new IllegalStateException("Strategy not initialized") + } + // If it is a new download check is a valid repository if( !localPath.exists() ) { - localPath.parentFile.mkdirs() - // make sure it contains a valid repository checkValidRemoteRepo(revision) - - final cloneURL = getGitRepositoryUrl() - log.debug "Pulling $project -- Using remote clone url: ${cloneURL}" - - // clone it, but don't specify a revision - jgit will checkout the default branch - def clone = Git.cloneRepository() - if( provider.hasCredentials() ) - clone.setCredentialsProvider( provider.getGitCredentials() ) - - clone - .setURI(cloneURL) - .setDirectory(localPath) - .setCloneSubmodules(manifest.recurseSubmodules) - if( deep ) - clone.setDepth(deep) - clone.call() - - // git cli would automatically create a 'refs/remotes/origin/HEAD' symbolic ref pointing at the remote's - // default branch. jgit doesn't do this, but since it automatically checked out the default branch on clone - // we can create the symbolic ref ourselves using the current head - def head = git.getRepository().findRef(Constants.HEAD) - if( head ) { - def headName = head.isSymbolic() - ? Repository.shortenRefName(head.getTarget().getName()) - : head.getName() - - git.repository.getRefDatabase() - .newUpdate(REMOTE_DEFAULT_HEAD, true) - .link(REMOTE_REFS_ROOT + headName) - } else { - log.debug "Unable to determine default branch of repo ${cloneURL}, symbolic ref not created" - } - - // now the default branch is recorded in the repo, explicitly checkout the revision (if specified). - // this also allows 'revision' to be a SHA commit id, which isn't supported by the clone command - if( revision ) { - try { git.checkout() .setName(revision) .call() } - catch ( RefNotFoundException e ) { checkoutRemoteBranch(revision) } - } - - // return status message - return "downloaded from ${cloneURL}" - } - - log.debug "Pull pipeline $project -- Using local path: $localPath" - - // verify that is clean - if( !isClean() ) - throw new AbortOperationException("$project contains uncommitted changes -- cannot pull from repository") - - if( revision && revision != getCurrentRevision() ) { - /* - * check out a revision before the pull operation - */ - try { - git.checkout() .setName(revision) .call() - } - /* - * If the specified revision does not exist - * Try to checkout it from a remote branch and return - */ - catch ( RefNotFoundException e ) { - final ref = checkoutRemoteBranch(revision) - final commitId = ref?.getObjectId() - return commitId - ? "checked out at ${commitId.name()}" - : "checked out revision ${revision}" - } - } - - def pull = git.pull() - def revInfo = getCurrentRevisionAndName() - - if ( revInfo.type == RevisionInfo.Type.COMMIT ) { - log.debug("Repo appears to be checked out to a commit hash, but not a TAG, so we will assume the repo is already up to date and NOT pull it!") - return MergeResult.MergeStatus.ALREADY_UP_TO_DATE.toString() - } - - if ( revInfo.type == RevisionInfo.Type.TAG ) { - pull.setRemoteBranchName( "refs/tags/" + revInfo.name ) - } - - if( provider.hasCredentials() ) - pull.setCredentialsProvider( provider.getGitCredentials() ) - - if( manifest.recurseSubmodules ) { - pull.setRecurseSubmodules(FetchRecurseSubmodulesMode.YES) } - def result = pull.call() - if(!result.isSuccessful()) - throw new AbortOperationException("Cannot pull project `$project` -- ${result.toString()}") - - return result?.mergeResult?.mergeStatus?.toString() - + return strategy.download(revision, deep, manifest) } /** @@ -718,46 +789,18 @@ class AssetManager { * @return The symbolic name of the current revision i.e. the current checked out branch or tag */ String getCurrentRevision() { - Ref head = git.getRepository().findRef(Constants.HEAD); - if( !head ) - return '(unknown)' - - if( head.isSymbolic() ) - return Repository.shortenRefName(head.getTarget().getName()) - - if( !head.getObjectId() ) - return '(unknown)' + if( !strategy ) { + throw new IllegalStateException("Strategy not initialized") + } + return strategy.getCurrentRevision() - // try to resolve the the current object id to a tag name - Map names = git.nameRev().addPrefix( "refs/tags/" ).add(head.objectId).call() - names.get( head.objectId ) ?: head.objectId.name() } RevisionInfo getCurrentRevisionAndName() { - Ref head = git.getRepository().findRef(Constants.HEAD); - if( !head ) - return null - - if( head.isSymbolic() ) { - return new RevisionInfo(head.objectId.name(), Repository.shortenRefName(head.getTarget().getName()), RevisionInfo.Type.BRANCH) - } - - if( !head.getObjectId() ) - return null - - // try to resolve the the current object id to a tag name - Map allNames = git.nameRev().addPrefix( "refs/tags/" ).add(head.objectId).call() - def name = allNames.get( head.objectId ) - if( name ) { - return new RevisionInfo(head.objectId.name(), name, RevisionInfo.Type.TAG) - } - else { - return new RevisionInfo(head.objectId.name(), null, RevisionInfo.Type.COMMIT) + if( !strategy ) { + throw new IllegalStateException("Strategy not initialized") } - } - - static boolean isRemoteBranch(Ref ref) { - return ref.name.startsWith(REMOTE_REFS_ROOT) && ref.name != REMOTE_DEFAULT_HEAD + return strategy.getCurrentRevisionAndName() } /** @@ -776,17 +819,17 @@ class AssetManager { @Deprecated List getRevisions(int level) { - def current = getCurrentRevision() + def pulled = strategy.listDownloadedCommits() def master = getDefaultBranch() List branches = getBranchList() .findAll { it.name.startsWith('refs/heads/') || isRemoteBranch(it) } .unique { shortenRefName(it.name) } - .collect { Ref it -> refToString(it,current,master,false,level) } + .collect { Ref it -> refToString(it, pulled, master, false, level) } List tags = getTagList() .findAll { it.name.startsWith('refs/tags/') } - .collect { refToString(it,current,master,true,level) } + .collect { refToString(strategy.peel(it) , pulled, master, true, level) } def result = new ArrayList(branches.size() + tags.size()) result.addAll(branches) @@ -807,68 +850,60 @@ class AssetManager { Map getBranchesAndTags(boolean checkForUpdates) { final result = [:] - final current = getCurrentRevision() + final currentCommits = strategy.listDownloadedCommits() final master = getDefaultBranch() final branches = [] final tags = [] - - Map remote = checkForUpdates ? git.lsRemote().callAsMap() : null + final pulled = new LinkedList() + log.debug("Current commits : $currentCommits") + Map remote = checkForUpdates ? strategy.lsRemote(false) : null getBranchList() - .findAll { it.name.startsWith('refs/heads/') || isRemoteBranch(it) } - .unique { shortenRefName(it.name) } - .each { Ref it -> branches << refToMap(it,remote) } + .findAll { it.name.startsWith('refs/heads/') || isRemoteBranch(it) } + .unique { shortenRefName(it.name) } + .each { + final map = refToMap(it, remote) + if( isRefInCommits(it, currentCommits) ) + pulled << map + branches << map + } - remote = checkForUpdates ? git.lsRemote().setTags(true).callAsMap() : null + remote = checkForUpdates ? strategy.lsRemote(true) : null getTagList() - .findAll { it.name.startsWith('refs/tags/') } - .each { Ref it -> tags << refToMap(it,remote) } + .findAll { it.name.startsWith('refs/tags/') } + .each { + Ref ref = strategy.peel(it) + final map = refToMap(ref, remote) + if( isRefInCommits(ref, currentCommits) ) + pulled << map + tags << map + } - result.current = current // current branch name + result.pulled = pulled.collect { it.name } // current pulled revisions result.master = master // master branch name result.branches = branches // collection of branches result.tags = tags // collect of tags return result } - protected Map refToMap(Ref ref, Map remote) { - final entry = new HashMap(2) - final peel = git.getRepository().getRefDatabase().peel(ref) - final objId = peel.getPeeledObjectId() ?: peel.getObjectId() - // the branch or tag name - entry.name = shortenRefName(ref.name) - // the local commit it - entry.commitId = objId.name() - // the remote commit Id for this branch or tag - if( remote && hasRemoteChange(ref,remote) ) { - entry.latestId = remote.get(ref.name).objectId.name() - } - return entry - } - @Memoized protected List getBranchList() { - git.branchList().setListMode(ListBranchCommand.ListMode.ALL) .call() + return strategy?.getBranchList() ?: [] } @Memoized protected List getTagList() { - git.tagList().call() + return strategy?.getTagList() ?: [] } - protected formatObjectId(ObjectId obj, boolean human) { - return human ? obj.name.substring(0,10) : obj.name - } - - protected String refToString(Ref ref, String current, String master, boolean tag, int level ) { + protected String refToString(Ref ref, List downloaded, String master, boolean tag, int level ) { def result = new StringBuilder() def name = shortenRefName(ref.name) - result << (name == current ? '*' : ' ') + result << ( isRefInCommits(ref, downloaded) ? '*' : ' ') if( level ) { - def peel = git.getRepository().getRefDatabase().peel(ref) - def obj = peel.getPeeledObjectId() ?: peel.getObjectId() + def obj = ref.getPeeledObjectId() ?: ref.getObjectId() result << ' ' result << formatObjectId(obj, level == 1) } @@ -883,41 +918,17 @@ class AssetManager { return result.toString() } - private String shortenRefName( String name ) { - if( name.startsWith('refs/remotes/origin/') ) - return name.replace('refs/remotes/origin/', '') - - return Repository.shortenRefName(name) - } - - protected String formatUpdate(Ref remoteRef, int level) { - - def result = new StringBuilder() - result << ' ' - result << formatObjectId(remoteRef.objectId, level<2) - result << ' ' - result << shortenRefName(remoteRef.name) - - return result.toString() - } - - protected hasRemoteChange(Ref ref, Map remote) { - if( !remote.containsKey(ref.name) ) - return false - ref.objectId.name != remote[ref.name].objectId.name - } - @Deprecated List getUpdates(int level) { - def remote = git.lsRemote().callAsMap() + def remote = strategy.lsRemote(false) List branches = getBranchList() .findAll { it.name.startsWith('refs/heads/') || isRemoteBranch(it) } .unique { shortenRefName(it.name) } .findAll { Ref ref -> hasRemoteChange(ref,remote) } .collect { Ref ref -> formatUpdate(remote.get(ref.name),level) } - remote = git.lsRemote().setTags(true).callAsMap() + remote = strategy.lsRemote(true) List tags = getTagList() .findAll { it.name.startsWith('refs/tags/') } .findAll { Ref ref -> hasRemoteChange(ref,remote) } @@ -930,29 +941,12 @@ class AssetManager { } /** - * Checkout a specific revision + * Checkout a specific revision, and fetch remote if not locally available. * @param revision The revision to be checked out */ - void checkout( String revision = null ) { - assert localPath - - def current = getCurrentRevision() - if( current != defaultBranch ) { - if( !revision ) { - throw new AbortOperationException("Project `$project` is currently stuck on revision: $current -- you need to explicitly specify a revision with the option `-r` in order to use it") - } - } - if( !revision || revision == current ) { - // nothing to do - return - } - - // verify that is clean - if( !isClean() ) - throw new AbortOperationException("Project `$project` contains uncommitted changes -- Cannot switch to revision: $revision") - + void checkout(String revision = null) { try { - git.checkout().setName(revision) .call() + tryCheckout(revision) } catch( RefNotFoundException e ) { checkoutRemoteBranch(revision) @@ -960,35 +954,25 @@ class AssetManager { } + /** + * Checkout a specific revision returns exception if revision is not found locally + * @param revision The revision to be checked out + */ + void tryCheckout(String revision = null) throws RefNotFoundException { + assert project + if( !strategy ) { + throw new IllegalStateException("Strategy not initialized") + } + strategy.tryCheckout(revision, manifest) + } - protected Ref checkoutRemoteBranch( String revision ) { - - try { - def fetch = git.fetch() - if(provider.hasCredentials()) { - fetch.setCredentialsProvider( provider.getGitCredentials() ) - } - if( manifest.recurseSubmodules ) { - fetch.setRecurseSubmodules(FetchRecurseSubmodulesMode.YES) - } - fetch.call() - try { - return git.checkout() - .setCreateBranch(true) - .setName(revision) - .setUpstreamMode(CreateBranchCommand.SetupUpstreamMode.TRACK) - .setStartPoint("origin/" + revision) - .call() - } - catch (RefNotFoundException e) { - return git.checkout() .setName(revision) .call() - } - } - catch (RefNotFoundException e) { - throw new AbortOperationException("Cannot find revision `$revision` -- Make sure that it exists in the remote repository `$repositoryUrl`", e) + protected Ref checkoutRemoteBranch(String revision) { + assert project + if( !strategy ) { + throw new IllegalStateException("Strategy not initialized") } - + return strategy.checkoutRemoteBranch(revision, manifest) } void updateModules() { @@ -1032,10 +1016,7 @@ class AssetManager { protected String getRemoteCommitId(RevisionInfo rev) { final tag = rev.type == RevisionInfo.Type.TAG - final cmd = git.lsRemote().setTags(tag) - if( provider.hasCredentials() ) - cmd.setCredentialsProvider( provider.getGitCredentials() ) - final list = cmd.call() + final list = strategy.lsRemote(tag).values() final ref = list.find { Repository.shortenRefName(it.name) == rev.name } if( !ref ) { log.debug "WARN: Cannot find any Git revision matching: ${rev.name}; ls-remote: $list" @@ -1072,7 +1053,7 @@ class AssetManager { } protected String getGitConfigRemoteUrl() { - if( !localPath ) { + if( !isLocal() ) { return null } @@ -1109,6 +1090,16 @@ class AssetManager { } + /** + * Drop local copy of a repository. If revision is specified, only removes the specified revision + * @param revision + */ + void drop(String revision = null, boolean force = false) { + if( isNotInitialized() ) + throw new AbortOperationException("No match found for: ${getProjectWithRevision()}") + strategy.drop(revision, force) + } + protected String guessHubProviderFromGitConfig(boolean failFast=false) { assert localPath @@ -1175,4 +1166,20 @@ class AssetManager { return commitId } } + + /** + * Enumeration of possible repository strategy types. + */ + enum RepositoryStrategyType { + /** + * Legacy full clone strategy + */ + LEGACY, + + /** + * Bare repo + Multi-revision strategy + */ + MULTI_REVISION + } + } diff --git a/modules/nextflow/src/main/groovy/nextflow/scm/GitReferenceHelper.groovy b/modules/nextflow/src/main/groovy/nextflow/scm/GitReferenceHelper.groovy new file mode 100644 index 0000000000..7a58d104d7 --- /dev/null +++ b/modules/nextflow/src/main/groovy/nextflow/scm/GitReferenceHelper.groovy @@ -0,0 +1,103 @@ +/* + * Copyright 2013-2025, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package nextflow.scm + +import groovy.transform.CompileStatic +import groovy.util.logging.Slf4j +import org.eclipse.jgit.lib.ObjectId +import org.eclipse.jgit.lib.Ref +import org.eclipse.jgit.lib.Repository + +/** + * Class with helper methods for Git references + * @author Jorge Ejarque + */ +@Slf4j +@CompileStatic +class GitReferenceHelper { + private static final String REMOTE_REFS_ROOT = "refs/remotes/origin/" + private static final String REMOTE_DEFAULT_HEAD = REMOTE_REFS_ROOT + "HEAD" + + /** + * Check if a ref has remote changes + */ + static boolean hasRemoteChange(Ref ref, Map remote) { + if( !remote ) + return false + + final remoteRef = remote.get(ref.name) + if( !remoteRef ) + return false + + return ref.getObjectId() != remoteRef.getObjectId() + } + + /** + * Format update information for a remote ref + */ + static String formatUpdate(Ref remoteRef, int level) { + final result = new StringBuilder() + result << 'updates on remote' + if( level ) { + result << ' ' + result << formatObjectId(remoteRef.getObjectId(), level == 1) + } + return result.toString() + } + + static boolean isRemoteBranch(Ref ref) { + return ref.name.startsWith(REMOTE_REFS_ROOT) && ref.name != REMOTE_DEFAULT_HEAD + } + + static formatObjectId(ObjectId obj, boolean human) { + return human ? obj.name.substring(0, 10) : obj.name + } + + static String shortenRefName(String name) { + if( name.startsWith('refs/remotes/origin/') ) + return name.replace('refs/remotes/origin/', '') + + return Repository.shortenRefName(name) + } + + static Map refToMap(Ref ref, Map remote) { + final entry = new HashMap(2) + final objId = ref.getPeeledObjectId() ?: ref.getObjectId() + // the branch or tag name + entry.name = shortenRefName(ref.name) + // the local commit it + entry.commitId = objId.name() + // the remote commit Id for this branch or tag + if( remote && hasRemoteChange(ref, remote) ) { + entry.latestId = remote.get(ref.name).objectId.name() + } + return entry + } + + static boolean isRefInCommits(Ref ref, List commits) { + if( !commits ) + return false + String peeledId = ref.getPeeledObjectId()?.name() + String id = ref.getObjectId()?.name() + for( String commit : commits ) { + if( commit.equals(peeledId) || commit.equals(id) ) + return true + } + return false + } + + +} diff --git a/modules/nextflow/src/main/groovy/nextflow/scm/GitlabRepositoryProvider.groovy b/modules/nextflow/src/main/groovy/nextflow/scm/GitlabRepositoryProvider.groovy index 68aed44da2..0e4bbc7132 100644 --- a/modules/nextflow/src/main/groovy/nextflow/scm/GitlabRepositoryProvider.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/scm/GitlabRepositoryProvider.groovy @@ -21,6 +21,8 @@ import groovy.util.logging.Slf4j import java.nio.charset.StandardCharsets +import static nextflow.Const.DEFAULT_BRANCH + /** * Implements a repository provider for GitHub service * @@ -71,8 +73,8 @@ class GitlabRepositoryProvider extends RepositoryProvider { String getDefaultBranch() { def result = invokeAndParseResponse(getEndpointUrl()) ?. default_branch if( !result ) { - log.debug "Unable to fetch repo default branch. Using `master` branch -- See https://gitlab.com/gitlab-com/support-forum/issues/1655#note_26132691" - return 'master' + log.debug "Unable to fetch repo default branch. Using `${DEFAULT_BRANCH}` branch -- See https://gitlab.com/gitlab-com/support-forum/issues/1655#note_26132691" + return DEFAULT_BRANCH } return result } diff --git a/modules/nextflow/src/main/groovy/nextflow/scm/LegacyRepositoryStrategy.groovy b/modules/nextflow/src/main/groovy/nextflow/scm/LegacyRepositoryStrategy.groovy new file mode 100644 index 0000000000..52baf7678b --- /dev/null +++ b/modules/nextflow/src/main/groovy/nextflow/scm/LegacyRepositoryStrategy.groovy @@ -0,0 +1,335 @@ +/* + * Copyright 2013-2025, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.scm + +import groovy.transform.CompileStatic +import groovy.transform.PackageScope +import groovy.util.logging.Slf4j +import nextflow.config.Manifest +import nextflow.exception.AbortOperationException +import org.eclipse.jgit.api.CreateBranchCommand +import org.eclipse.jgit.api.Git +import org.eclipse.jgit.api.ListBranchCommand +import org.eclipse.jgit.api.MergeResult +import org.eclipse.jgit.api.errors.RefNotFoundException +import org.eclipse.jgit.errors.RepositoryNotFoundException +import org.eclipse.jgit.lib.Constants +import org.eclipse.jgit.lib.Ref +import org.eclipse.jgit.lib.Repository +import org.eclipse.jgit.lib.SubmoduleConfig.FetchRecurseSubmodulesMode + +/** + * Legacy repository strategy that directly clones repositories to the local path. + * This is the traditional approach where each project gets a full git clone. + * + * @author Jorge Ejarque + */ +@Slf4j +@CompileStatic +class LegacyRepositoryStrategy extends AbstractRepositoryStrategy { + + private Git _git + private File localPath + + LegacyRepositoryStrategy(String project) { + super(project) + if( project ) + this.localPath = new File(root, project) + } + + @Override + void setProject(String project) { + super.setProject(project) + this.localPath = new File(root, project) + } + + @PackageScope + static boolean checkProject(File root, String project) { + return new File(root, project + '/.git').exists() + } + + @Override + String download(String revision, Integer deep, Manifest manifest) { + assert localPath + /* + * if the pipeline already exists locally pull it from the remote repo + */ + if( !localPath.exists() ) { + assert provider + getLocalPath().parentFile.mkdirs() + + final cloneURL = getGitRepositoryUrl() + log.debug "Pulling ${project} -- Using remote clone url: ${cloneURL}" + + // clone it, but don't specify a revision - jgit will checkout the default branch + def clone = Git.cloneRepository() + if( provider.hasCredentials() ) + clone.setCredentialsProvider(provider.getGitCredentials()) + + clone + .setURI(cloneURL) + .setDirectory(getLocalPath()) + .setCloneSubmodules(manifest.recurseSubmodules) + if( deep ) + clone.setDepth(deep) + clone.call() + + // git cli would automatically create a 'refs/remotes/origin/HEAD' symbolic ref pointing at the remote's + // default branch. jgit doesn't do this, but since it automatically checked out the default branch on clone + // we can create the symbolic ref ourselves using the current head + def head = getGit().getRepository().findRef(Constants.HEAD) + if( head ) { + def headName = head.isSymbolic() + ? Repository.shortenRefName(head.getTarget().getName()) + : head.getName() + + getGit().repository.getRefDatabase() + .newUpdate(REMOTE_DEFAULT_HEAD, true) + .link(REMOTE_REFS_ROOT + headName) + } else { + log.debug "Unable to determine default branch of repo ${cloneURL}, symbolic ref not created" + } + + // now the default branch is recorded in the repo, explicitly checkout the revision (if specified). + // this also allows 'revision' to be a SHA commit id, which isn't supported by the clone command + if( revision ) { + try { + getGit().checkout().setName(revision).call() + } + catch( RefNotFoundException e ) { + checkoutRemoteBranch(revision, manifest) + } + } + + // return status message + return "downloaded from ${cloneURL}" + } + + log.debug "Pull pipeline ${project} -- Using local path: ${getLocalPath()}" + + // verify that is clean + if( !isClean() ) + throw new AbortOperationException("${project} contains uncommitted changes -- cannot pull from repository") + + if( revision && revision != getCurrentRevision() ) { + /* + * check out a revision before the pull operation + */ + try { + getGit().checkout().setName(revision).call() + } + /* + * If the specified revision does not exist + * Try to checkout it from a remote branch and return + */ + catch( RefNotFoundException e ) { + final ref = checkoutRemoteBranch(revision, manifest) + final commitId = ref?.getObjectId() + return commitId + ? "checked out at ${commitId.name()}" + : "checked out revision ${revision}" + } + } + + def pull = getGit().pull() + def revInfo = getCurrentRevisionAndName() + + if( revInfo.type == AssetManager.RevisionInfo.Type.COMMIT ) { + log.debug("Repo appears to be checked out to a commit hash, but not a TAG, so we will assume the repo is already up to date and NOT pull it!") + return MergeResult.MergeStatus.ALREADY_UP_TO_DATE.toString() + } + + if( revInfo.type == AssetManager.RevisionInfo.Type.TAG ) { + pull.setRemoteBranchName("refs/tags/" + revInfo.name) + } + + if( provider.hasCredentials() ) + pull.setCredentialsProvider(provider.getGitCredentials()) + + if( manifest.recurseSubmodules ) { + pull.setRecurseSubmodules(FetchRecurseSubmodulesMode.YES) + } + def result = pull.call() + if( !result.isSuccessful() ) + throw new AbortOperationException("Cannot pull project `${project}` -- ${result.toString()}") + + return result?.mergeResult?.mergeStatus?.toString() + } + + @Override + void tryCheckout(String revision, Manifest manifest) throws RefNotFoundException { + assert localPath + + def current = getCurrentRevision() + if( current != getDefaultBranch(manifest) ) { + if( !revision ) { + throw new AbortOperationException("Project `$project` is currently stuck on revision: $current -- you need to explicitly specify a revision with the option `-r` in order to use it") + } + } + if( !revision || revision == current ) { + // nothing to do + return + } + + // verify that is clean + if( !isClean() ) + throw new AbortOperationException("Project `$project` contains uncommitted changes -- Cannot switch to revision: $revision") + + + git.checkout().setName(revision).call() + } + + @Override + Ref checkoutRemoteBranch(String revision, Manifest manifest) { + assert provider + try { + def fetch = git.fetch() + if( provider.hasCredentials() ) { + fetch.setCredentialsProvider(provider.getGitCredentials()) + } + if( manifest.recurseSubmodules ) { + fetch.setRecurseSubmodules(FetchRecurseSubmodulesMode.YES) + } + fetch.call() + + try { + return git.checkout() + .setCreateBranch(true) + .setName(revision) + .setUpstreamMode(CreateBranchCommand.SetupUpstreamMode.TRACK) + .setStartPoint("origin/" + revision) + .call() + } + catch( RefNotFoundException e ) { + return git.checkout().setName(revision).call() + } + } + catch( RefNotFoundException e ) { + throw new AbortOperationException("Cannot find revision `$revision` -- Make sure that it exists in the remote repository `$gitRepositoryUrl`", e) + } + } + + @Override + File getLocalPath() { + return this.localPath + } + + @Override + void setLocalPath(File file) { + this.localPath = file + if( _git ) + _git.close() + } + + @Override + File getProjectPath() { + return this.localPath + } + + @Override + Git getGit() { + if( !_git ) { + _git = Git.open(localPath) + } + return _git + } + + @Override + boolean isClean() { + try { + getGit().status().call().isClean() + } + catch( RepositoryNotFoundException e ) { + return true + } + } + + @Override + String getRemoteDefaultBranch() { + Ref remoteHead = git.getRepository().findRef(REMOTE_DEFAULT_HEAD) + return remoteHead?.getTarget()?.getName()?.substring(REMOTE_REFS_ROOT.length()) + } + + @Override + List getBranchList() { + getGit().branchList().setListMode(ListBranchCommand.ListMode.ALL).call() + } + + @Override + List getTagList() { + getGit().tagList().call() + } + + @Override + Map lsRemote(boolean tags) { + assert provider + final cmd = getGit().lsRemote().setTags(tags) + if( provider.hasCredentials() ) + cmd.setCredentialsProvider(provider.getGitCredentials()) + return cmd.callAsMap() + } + + @Override + Ref peel(Ref ref) { + return getGit().getRepository().getRefDatabase().peel(ref) + } + + @Override + File getLocalGitConfig() { + getLocalPath() ? new File(getLocalPath(), '.git/config') : null + } + + @Override + String getGitRepositoryUrl() { + if( localPath.exists() ) { + return localPath.toURI().toString() + } + return provider.getCloneUrl() + } + + @Override + List listDownloadedCommits() { + if( !localPath.exists() ) + return [] + return [findHeadRef().objectId.getName()] + } + + @Override + void drop(String revision, boolean force) { + if( !localPath.exists() ) + throw new AbortOperationException("No match found for: ${project}") + + if( revision ) + throw new AbortOperationException("Not able to remove a revision for a Legacy Repo. Use all option to remove the local repository.") + + if( force || isClean() ) { + close() + if( !localPath.deleteDir() ) + throw new AbortOperationException("Unable to delete project `${project}` -- Check access permissions for path: ${localPath}") + return + } + throw new AbortOperationException("Local project repository contains uncommitted changes -- won't drop it") + } + + @Override + void close() { + if( _git ) { + _git.close() + _git = null + } + } +} diff --git a/modules/nextflow/src/main/groovy/nextflow/scm/MultiRevisionRepositoryStrategy.groovy b/modules/nextflow/src/main/groovy/nextflow/scm/MultiRevisionRepositoryStrategy.groovy new file mode 100644 index 0000000000..0cc458ea59 --- /dev/null +++ b/modules/nextflow/src/main/groovy/nextflow/scm/MultiRevisionRepositoryStrategy.groovy @@ -0,0 +1,510 @@ +/* + * Copyright 2013-2025, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.scm + +import groovy.transform.CompileStatic +import groovy.transform.PackageScope +import groovy.util.logging.Slf4j +import nextflow.config.Manifest +import nextflow.exception.AbortOperationException +import nextflow.file.FileMutex +import org.eclipse.jgit.api.Git +import org.eclipse.jgit.api.ListBranchCommand +import org.eclipse.jgit.api.errors.RefNotFoundException +import org.eclipse.jgit.errors.RepositoryNotFoundException +import org.eclipse.jgit.internal.storage.file.FileRepository +import org.eclipse.jgit.lib.Constants +import org.eclipse.jgit.lib.Ref +import org.eclipse.jgit.lib.Repository +import org.eclipse.jgit.lib.SubmoduleConfig +import org.eclipse.jgit.storage.file.FileRepositoryBuilder +import org.eclipse.jgit.transport.RefSpec + +/** + * Multi-revision repository strategy that uses a bare repository with shared clones. + * This approach allows multiple revisions to coexist efficiently by sharing objects + * through a bare repository and creating lightweight clones for each commit. + * + * @author Jorge Ejarque + */ +@Slf4j +@CompileStatic +class MultiRevisionRepositoryStrategy extends AbstractRepositoryStrategy { + + static final String REPOS_SUBDIR = '.repos' + static final String BARE_REPO = 'bare' + static final String REVISION_SUBDIR = 'commits' + + /** + * The revision (branch, tag, or commit SHA) to work with + */ + private String revision + + /** + * Path to the commit-specific clone + */ + private File legacyRepoPath + private File projectPath + private File commitPath + private File bareRepo + private File revisionSubdir + private Git _bareGit + private Git _commitGit + + MultiRevisionRepositoryStrategy(String project, String revision = null) { + super(project) + if( project ) { + this.legacyRepoPath = new File(root, project) + this.projectPath = new File(root, REPOS_SUBDIR + '/' + project) + this.bareRepo = new File(projectPath, BARE_REPO) + this.revisionSubdir = new File(projectPath, REVISION_SUBDIR) + } + if( revision ) + setRevision(revision) + else if( hasBareRepo() ) { + setRevision(getDefaultBranchFromBareRepo()) + } + + } + /** + * Check if a project exists and has the bare repo for multi-revision + * @param root + * @param project + * @return + */ + static boolean checkProject(File root, String project) { + return new File(root, REPOS_SUBDIR + '/' + project + '/' + BARE_REPO).exists() + } + + @PackageScope + File getBareRepo() { this.bareRepo } + + boolean hasBareRepo() { + return this.bareRepo && this.bareRepo.exists() + } + + @PackageScope + File getRevisionSubdir() { this.revisionSubdir } + + @PackageScope + void setRevision(String revision) { + assert revision + this.revision = revision + updateCommitDir(revisionToCommitWithBareRepo(revision)) + } + + String getRevision() { + return revision + } + + @PackageScope + void updateCommitDir(String commitId) { + final oldCommitPath = this.commitPath + if( oldCommitPath && _commitGit ) { + _commitGit.close() + _commitGit = null + } + this.commitPath = commitId ? new File(getRevisionSubdir(), commitId) : null + } + + @PackageScope + String revisionToCommitWithBareRepo(String revision) { + String commitId = null + + if( hasBareRepo() ) { + def rev = Git.open(this.bareRepo) + .getRepository() + .resolve(revision ?: Constants.HEAD) + if( rev ) + commitId = rev.getName() + } + + return commitId + } + + protected String getDefaultBranchFromBareRepo() { + if( !hasBareRepo() ) { + log.debug "Bare repo ${this.bareRepo} doesn't exist" + return null + } + def head = getDefaultBranchRef() + if( head ) { + Repository.shortenRefName(head.getName()) + } else { + log.debug "Unable to determine default branch from bare repo ${this.bareRepo}" + return null + } + } + + protected Ref getDefaultBranchRef() { + return getBareGit().getRepository().findRef(Constants.HEAD).getTarget() + } + + protected void checkBareRepo(Manifest manifest) { + assert bareRepo + /* + * if the bare repository of the pipeline does not exists locally pull it from the remote repo + */ + if( !hasBareRepo() ) { + if( !bareRepo.parentFile.exists() ) + this.bareRepo.parentFile.mkdirs() + // Use a file mutex to prevent concurrent clones of the same commit. + final file = new File(this.bareRepo.parentFile, ".${this.bareRepo.name}.lock") + final wait = "Another Nextflow instance is creating the bare repo for ${project} -- please wait till it completes" + final err = "Unable to acquire exclusive lock after 60s on file: $file" + + final mutex = new FileMutex(target: file, timeout: '60s', waitMessage: wait, errorMessage: err) + try { + mutex.lock { createBareRepo(manifest) } + } + finally { + file.delete() + } + } + final updateRevision = revision ?: getDefaultBranch(manifest) + log.debug "Fetching (updating) bare repo for ${project} [revision: $updateRevision]" + getBareGit().fetch().setRefSpecs(refSpecForName(updateRevision)).call() + } + + private void createBareRepo(Manifest manifest) { + assert provider + // This check is required in case of two nextflow instances were doing the same shared clone at the same time. + // If commitPath exists the previous nextflow instance created the shared clone successfully. + if( hasBareRepo() ) + return + + try { + final cloneURL = provider.getCloneUrl() + log.debug "Pulling bare repo for ${project} -- Using remote clone url: ${cloneURL}" + def bare = Git.cloneRepository() + if( provider.hasCredentials() ) + bare.setCredentialsProvider(provider.getGitCredentials()) + + bare + .setBare(true) + .setURI(cloneURL) + .setGitDir(this.bareRepo) + .setCloneSubmodules(manifest.recurseSubmodules) + .call() + } catch( Throwable t ) { + // If there is an error creating the bare repo, remove the bare repo path to avoid incorrect repo clones. + bareRepo.deleteDir() + throw t + } + } + + @Override + String download(String revision, Integer deep, Manifest manifest) { + + // Update revision if specified + if( revision ) + setRevision(revision) + + // get local copy of bare repository if not exists and fetch revision if specified + checkBareRepo(manifest) + + // Try to get the commit Id for the revision and abort if not found. + this.revision ?= getDefaultBranch(manifest) + final commitId = revisionToCommitWithBareRepo(this.revision) + if( !commitId ) { + throw new AbortOperationException("No commit found for revision ${this.revision} in project ${project}") + } + updateCommitDir(commitId) + + if( commitPath.exists() ) + return "Already-up-to-date" + + /* + * if revision does not exists locally pull it from the remote repo + */ + if( !commitPath.parentFile.exists() ) + commitPath.parentFile.mkdirs() + + // Use a file mutex to prevent concurrent clones of the same commit. + final file = new File(commitPath.parentFile, ".${commitPath.name}.lock") + final wait = "Another Nextflow instance is creating clone for $this.revision -- please wait till it completes" + final err = "Unable to acquire exclusive lock after 60s on file: $file" + + final mutex = new FileMutex(target: file, timeout: '60s', waitMessage: wait, errorMessage: err) + try { + mutex.lock { createSharedClone(this.revision, manifest.recurseSubmodules) } + } + finally { + file.delete() + } + return "downloaded from ${getGitRepositoryUrl()}" + } + + /** Checkout in multi-revision is just updating the commit dir. This methods check if bare repo, commit id and commit path exists. + * If exists updates the revision that updates the localPath reference. If some + * + * @param revision + * @param recurseSubmodules + * @throws RefNotFoundException + */ + @Override + void tryCheckout(String revision, Manifest manifest) throws RefNotFoundException { + if( !hasBareRepo() ) { + throw new RefNotFoundException("Unknown repository") + } + // get the commit ID for the revision. If not specified try to checkout to default branch + final downloadRevision = revision ?: getDefaultBranch(manifest) + final commitId = revisionToCommitWithBareRepo(downloadRevision) + if( !commitId ) { + throw new RefNotFoundException("No commit found for revision ${downloadRevision} in project ${project}") + } + if( isRevisionLocal(commitId) ) { + setRevision(downloadRevision) + } else { + throw new RefNotFoundException("Revision ${downloadRevision} not locally checked out.") + } + } + + private boolean isRevisionLocal(String commitId) { + this.revisionSubdir && new File(revisionSubdir, commitId).exists() + } + + @Override + Ref checkoutRemoteBranch(String revision, Manifest manifest) { + download(revision, 1, manifest) + return findHeadRef() + } + + private String createSharedClone(String downloadRevision, boolean recurseSubmodules) { + // This check is required in case of two nextflow instances were doing the same shared clone at the same time. + // If commitPath exists the previous nextflow instance created the shared clone successfully. + if( commitPath.exists() ) + return "Already-up-to-date" + + commitPath.parentFile.mkdirs() + try { + final cloneURL = this.bareRepo.toString() + log.debug "Pulling ${project} -- Using remote clone url: ${cloneURL}" + + // clone it, but don't specify a revision - jgit will checkout the default branch + File bareObjectsDir = new File(this.bareRepo, "objects") + FileRepository repo = new FileRepositoryBuilder() + .setGitDir(new File(getLocalPath(), ".git")) + .addAlternateObjectDirectory(bareObjectsDir) + .build() as FileRepository + repo.create() + + // Write alternates file (not done by repo create). + new File(repo.getObjectsDirectory(), "info/alternates").write(bareObjectsDir.absolutePath) + + // Configure remote pointing to the cache repo + repo.getConfig().setString("remote", "origin", "url", cloneURL) + repo.getConfig().save() + + final fetch = getCommitGit() + .fetch() + .setRefSpecs(refSpecForName(downloadRevision)) + if( recurseSubmodules ) + fetch.setRecurseSubmodules(SubmoduleConfig.FetchRecurseSubmodulesMode.YES) + fetch.call() + getCommitGit().checkout().setName(downloadRevision).call() + + } catch( Throwable t ) { + // If there is an error creating the shared clone, remove the local path to avoid incorrect clones. + commitPath.deleteDir() + throw t + } + + // return status message + return "downloaded from ${getGitRepositoryUrl()}" + } + + private RefSpec refSpecForName(String revision) { + // Is it a local branch? + Ref branch = getBareGit().getRepository().findRef("refs/heads/" + revision) + if( branch != null ) { + return new RefSpec("refs/heads/" + revision + ":refs/heads/" + revision) + } + + // Is it a tag? + Ref tag = getBareGit().getRepository().findRef("refs/tags/" + revision) + if( tag != null ) { + return new RefSpec("refs/tags/" + revision + ":refs/tags/" + revision) + } + + // It is a commit + return new RefSpec(revision + ":refs/tags/" + revision) + } + + @Override + File getLocalPath() { + return this.commitPath ?: legacyRepoPath + } + + protected Git getBareGit() { + assert bareRepo + if( !_bareGit ) { + _bareGit = Git.open(this.bareRepo) + } + return _bareGit + } + + protected Git getCommitGit() { + if( !_commitGit ) { + _commitGit = Git.open(commitPath) + } + return _commitGit + } + + @Override + Git getGit() { + if( commitPath && commitPath.exists() ) + return getCommitGit() + if( hasBareRepo() ) + return getBareGit() + if( this.legacyRepoPath && new File(this.legacyRepoPath, '.git').exists() ) + return Git.open(this.legacyRepoPath) + return null + } + + private Git getBareGitWithLegacyFallback() { + if( hasBareRepo() ) + return getBareGit() + // Fallback to legacy + if( this.legacyRepoPath && new File(this.legacyRepoPath, '.git').exists() ) + return Git.open(this.legacyRepoPath) + return null + } + + @Override + boolean isClean() { + try { + getCommitGit().status().call().isClean() + } + catch( RepositoryNotFoundException e ) { + return true + } + } + + @Override + String getRemoteDefaultBranch() { + return getDefaultBranchFromBareRepo() ?: findRemoteDefaultBranch() + } + + String findRemoteDefaultBranch() { + Ref remoteHead = git.getRepository().findRef(REMOTE_DEFAULT_HEAD) + return remoteHead?.getTarget()?.getName()?.substring(REMOTE_REFS_ROOT.length()) + } + + @Override + List getBranchList() { + getBareGitWithLegacyFallback()?.branchList()?.setListMode(ListBranchCommand.ListMode.ALL)?.call() ?: [] + } + + @Override + List getTagList() { + getBareGitWithLegacyFallback()?.tagList()?.call() ?: [] + } + + @Override + Map lsRemote(boolean tags) { + final cmd = getBareGitWithLegacyFallback()?.lsRemote()?.setTags(tags) + if( provider.hasCredentials() ) + cmd?.setCredentialsProvider(provider.getGitCredentials()) + return cmd?.callAsMap() ?: [:] + } + + @Override + Ref peel(Ref ref) { + return getBareGitWithLegacyFallback()?.getRepository()?.getRefDatabase()?.peel(ref) + } + + @Override + File getLocalGitConfig() { + return hasBareRepo() ? new File(this.bareRepo, 'config') : legacyRepoPath ? new File(legacyRepoPath,'.git/config') : null + } + + @Override + String getGitRepositoryUrl() { + return provider.getCloneUrl() + } + + @Override + List listDownloadedCommits() { + def result = new LinkedList() + if( !getRevisionSubdir().exists() ) + return result + + getRevisionSubdir().eachDir { File it -> result << it.getName().toString() } + + return result + } + + @Override + void drop(String revision, boolean force) { + if( revision ) { + dropRevision(revision, force) + } else { + listDownloadedCommits().each { dropRevision(it, force) } + bareRepo.parentFile.deleteDir() + } + } + + private void dropRevision(String revision, boolean force) { + assert revision + setRevision(revision) + + if( !commitPath || !commitPath.exists() ) { + log.info "No local folder found for revision '$revision' in '$project' -- Nothing to do" + return + } + + if( force || isClean() ) { + close() + if( !commitPath.deleteDir() ) + throw new AbortOperationException("Unable to delete project `${project}` -- Check access permissions for path: ${localPath}") + return + } + + throw new AbortOperationException("Revision $revision for project `${project}` contains uncommitted changes -- won't drop it") + } + + @Override + void close() { + if( _bareGit ) { + _bareGit.close() + _bareGit = null + } + if( _commitGit ) { + _commitGit.close() + _commitGit = null + } + } + + @Override + void setLocalPath(File file) { + legacyRepoPath = file + } + + @Override + void setProject(String project) { + super.setProject(project) + this.projectPath = new File(root, REPOS_SUBDIR + '/' + project) + this.bareRepo = new File(projectPath, BARE_REPO) + this.revisionSubdir = new File(projectPath, REVISION_SUBDIR) + } + + @Override + File getProjectPath() { + return projectPath + } +} diff --git a/modules/nextflow/src/main/groovy/nextflow/scm/RepositoryStrategy.groovy b/modules/nextflow/src/main/groovy/nextflow/scm/RepositoryStrategy.groovy new file mode 100644 index 0000000000..66b82c7cf1 --- /dev/null +++ b/modules/nextflow/src/main/groovy/nextflow/scm/RepositoryStrategy.groovy @@ -0,0 +1,161 @@ +/* + * Copyright 2013-2025, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.scm + +import groovy.transform.CompileStatic +import nextflow.config.Manifest +import org.eclipse.jgit.api.Git +import org.eclipse.jgit.api.errors.RefNotFoundException +import org.eclipse.jgit.lib.Ref + +/** + * Strategy interface for different repository management approaches. + * Implementations handle the specifics of how pipelines are stored and accessed locally. + * + * @author Jorge Ejarque + */ +@CompileStatic +interface RepositoryStrategy { + + /** + * Download or update the repository. + * For legacy strategy, this clones the repo directly to the local path. + * For multi-revision strategy, this creates a bare repo and shared clones. + * + * @param revision The revision to download (branch, tag, or commit SHA) + * @param deep Optional depth for shallow clones + * @return Status message describing what was done + */ + String download(String revision, Integer deep, Manifest manifest) + + /** + * Checkout a specific revision and returns exception if revision is not found locally + * @param revision The revision to be checked out + */ + void tryCheckout(String revision, Manifest manifest) throws RefNotFoundException + + /** + * Fetch and checkout a specific revision + * @param revision The revision to be checked out + */ + Ref checkoutRemoteBranch(String revision, Manifest manifest) + + /** + * @return The local path where the worktree and repository files are accessible + */ + File getLocalPath() + + /** + * @return The Git repository object for repository operations + */ + Git getGit() + + /** + * @return True if the working directory has no uncommitted changes + */ + boolean isClean() + + /** + * @return The default branch name from the remote repository + */ + String getRemoteDefaultBranch() + + /** + * @return List of all branches in the repository + */ + List getBranchList() + + /** + * @return List of all tags in the repository + */ + List getTagList() + + /** + * Query remote repository for refs + * + * @param tags If true, query for tags; otherwise query for branches + * @return Map of ref names to Ref objects + */ + Map lsRemote(boolean tags) + + /** + * Peel a ref (resolves annotated tags to their target) + * + * @param ref The ref to peel + * @return The peeled ref + */ + Ref peel(Ref ref) + + /** + * @return The local git config file + */ + File getLocalGitConfig() + + /** + * @return The local asset repository URL + */ + String getGitRepositoryUrl() + + /** + * @return The current revision in the local repository + */ + String getCurrentRevision() + + /** + * @return The list of current downloaded commits in the local repository + */ + List listDownloadedCommits() + + /** + * @return The current revision and Name + */ + AssetManager.RevisionInfo getCurrentRevisionAndName() + + /** + * Drop local copy of a repository. If revision is specified, only removes the specified revision + * @param revision + */ + void drop(String revision, boolean force) + + /** + * Close any open resources (e.g., Git objects) + */ + void close() + + /** + * Set the localPath + * @param file + */ + void setLocalPath(File file) + + /** + * Set project + * @param projectName + */ + void setProject(String projectName) + + /** + * @return return the local path where project repository and revisions are download + */ + File getProjectPath() + + /** + * Set repository provider + * @param provider + */ + void setProvider(RepositoryProvider provider) +} diff --git a/modules/nextflow/src/test/groovy/nextflow/cli/CmdInfoTest.groovy b/modules/nextflow/src/test/groovy/nextflow/cli/CmdInfoTest.groovy index e8949c946d..7eaf6e91f7 100644 --- a/modules/nextflow/src/test/groovy/nextflow/cli/CmdInfoTest.groovy +++ b/modules/nextflow/src/test/groovy/nextflow/cli/CmdInfoTest.groovy @@ -16,6 +16,8 @@ package nextflow.cli +import static nextflow.scm.MultiRevisionRepositoryStrategy.REPOS_SUBDIR + import nextflow.plugin.Plugins import spock.lang.IgnoreIf import spock.lang.Requires @@ -45,10 +47,11 @@ class CmdInfoTest extends Specification { def setupSpec() { tempDir = Files.createTempDirectory('test') AssetManager.root = tempDir.toFile() + String revision = null def token = System.getenv('NXF_GITHUB_ACCESS_TOKEN') def manager = new AssetManager().build('nextflow-io/hello', [providers: [github: [auth: token]]]) // download the project - manager.download() + manager.download(revision) } def cleanupSpec() { @@ -67,7 +70,7 @@ class CmdInfoTest extends Specification { then: screen.contains(" project name: nextflow-io/hello") screen.contains(" repository : https://github.com/nextflow-io/hello") - screen.contains(" local path : $tempDir/nextflow-io/hello" ) + screen.contains(" local path : $tempDir/$REPOS_SUBDIR/nextflow-io/hello" ) screen.contains(" main script : main.nf") screen.contains(" revisions : ") screen.contains(" * master (default)") @@ -88,11 +91,12 @@ class CmdInfoTest extends Specification { then: json.projectName == "nextflow-io/hello" json.repository == "https://github.com/nextflow-io/hello" - json.localPath == "$tempDir/nextflow-io/hello" + json.localPath == "${tempDir}/${REPOS_SUBDIR}/nextflow-io/hello".toString() json.manifest.mainScript == 'main.nf' json.manifest.defaultBranch == null - json.revisions.current == 'master' json.revisions.master == 'master' + json.revisions.pulled.size() == 1 + json.revisions.pulled.any { it == 'master' } json.revisions.branches.size()>1 json.revisions.branches.any { it.name == 'master' } !json.revisions.branches.any { it.name == 'HEAD' } @@ -115,11 +119,12 @@ class CmdInfoTest extends Specification { then: json.projectName == "nextflow-io/hello" json.repository == "https://github.com/nextflow-io/hello" - json.localPath == "$tempDir/nextflow-io/hello" + json.localPath == "${tempDir}/${REPOS_SUBDIR}/nextflow-io/hello" json.manifest.mainScript == 'main.nf' json.manifest.defaultBranch == null - json.revisions.current == 'master' json.revisions.master == 'master' + json.revisions.pulled.size() == 1 + json.revisions.pulled.any { it == 'master' } json.revisions.branches.size()>1 json.revisions.branches.any { it.name == 'master' } !json.revisions.branches.any { it.name == 'HEAD' } diff --git a/modules/nextflow/src/test/groovy/nextflow/cli/CmdPullTest.groovy b/modules/nextflow/src/test/groovy/nextflow/cli/CmdPullTest.groovy index de3fea6853..ebfc799ed3 100644 --- a/modules/nextflow/src/test/groovy/nextflow/cli/CmdPullTest.groovy +++ b/modules/nextflow/src/test/groovy/nextflow/cli/CmdPullTest.groovy @@ -16,6 +16,9 @@ package nextflow.cli +import static nextflow.scm.MultiRevisionRepositoryStrategy.REPOS_SUBDIR +import static nextflow.scm.MultiRevisionRepositoryStrategy.REVISION_SUBDIR + import nextflow.plugin.Plugins import spock.lang.IgnoreIf @@ -40,13 +43,13 @@ class CmdPullTest extends Specification { given: def accessToken = System.getenv('NXF_GITHUB_ACCESS_TOKEN') def dir = Files.createTempDirectory('test') - def cmd = new CmdPull(args: ['nextflow-io/hello'], root: dir.toFile(), hubUser: accessToken) + def cmd = new CmdPull(args: ['nextflow-io/hello'], root: dir.toFile(), revision: '7588c46ffefb4e3c06d4ab32c745c4d5e56cdad8', hubUser: accessToken) when: cmd.run() then: - dir.resolve('nextflow-io/hello/.git').exists() - dir.resolve('nextflow-io/hello/README.md').exists() + dir.resolve(REPOS_SUBDIR + '/nextflow-io/hello/' + REVISION_SUBDIR + '/' + '7588c46ffefb4e3c06d4ab32c745c4d5e56cdad8' + '/.git').exists() + dir.resolve(REPOS_SUBDIR+ '/nextflow-io/hello/' + REVISION_SUBDIR + '/' + '7588c46ffefb4e3c06d4ab32c745c4d5e56cdad8' + '/README.md').exists() cleanup: dir?.deleteDir() diff --git a/modules/nextflow/src/test/groovy/nextflow/scm/AssetManagerTest.groovy b/modules/nextflow/src/test/groovy/nextflow/scm/AssetManagerTest.groovy index ac59065c25..7391ab0e23 100644 --- a/modules/nextflow/src/test/groovy/nextflow/scm/AssetManagerTest.groovy +++ b/modules/nextflow/src/test/groovy/nextflow/scm/AssetManagerTest.groovy @@ -16,6 +16,13 @@ package nextflow.scm +import java.util.concurrent.CountDownLatch +import java.util.concurrent.Executors + +import static MultiRevisionRepositoryStrategy.BARE_REPO +import static MultiRevisionRepositoryStrategy.REVISION_SUBDIR +import static nextflow.scm.MultiRevisionRepositoryStrategy.REPOS_SUBDIR + import spock.lang.IgnoreIf import nextflow.exception.AbortOperationException @@ -25,7 +32,6 @@ import org.junit.Rule import spock.lang.Requires import spock.lang.Specification import test.TemporaryPath -import java.nio.file.Path import java.nio.file.Paths /** * @@ -71,7 +77,7 @@ class AssetManagerTest extends Specification { AssetManager.root = tempDir.root.toFile() } - // Helper method to grab the default brasnch if set in ~/.gitconfig + // Helper method to grab the default branch if set in ~/.gitconfig String getLocalDefaultBranch() { def defaultBranch = 'master' def gitconfig = Paths.get(System.getProperty('user.home'),'.gitconfig'); @@ -90,16 +96,19 @@ class AssetManagerTest extends Specification { folder.resolve('cbcrg/pipe1').mkdirs() folder.resolve('cbcrg/pipe2').mkdirs() folder.resolve('ncbi/blast').mkdirs() + folder.resolve(REPOS_SUBDIR +'/ncbi/blast').mkdirs() + folder.resolve(REPOS_SUBDIR +'/new/repo').mkdirs() when: def list = AssetManager.list() then: - list.sort() == ['cbcrg/pipe1','cbcrg/pipe2','ncbi/blast'] + list.sort() == ['cbcrg/pipe1','cbcrg/pipe2','ncbi/blast', 'new/repo'] expect: AssetManager.find('blast') == 'ncbi/blast' AssetManager.find('pipe1') == 'cbcrg/pipe1' AssetManager.find('pipe') as Set == ['cbcrg/pipe1', 'cbcrg/pipe2'] as Set + AssetManager.find('repo') == 'new/repo' } @@ -156,14 +165,14 @@ class AssetManagerTest extends Specification { } - @Requires({System.getenv('NXF_GITHUB_ACCESS_TOKEN')}) - def testPull() { + def 'test download with legacy'() { given: def folder = tempDir.getRoot() def token = System.getenv('NXF_GITHUB_ACCESS_TOKEN') def manager = new AssetManager().build('nextflow-io/hello', [providers: [github: [auth: token]]]) + manager.setStrategyType(AssetManager.RepositoryStrategyType.LEGACY) when: manager.download() @@ -177,19 +186,42 @@ class AssetManagerTest extends Specification { } + def 'test download from tag twice with multi-revision'() { + + given: + def folder = tempDir.getRoot() + def token = System.getenv('NXF_GITHUB_ACCESS_TOKEN') + def manager = new AssetManager().build('nextflow-io/hello', [providers: [github: [auth: token]]]) + manager.setStrategyType(AssetManager.RepositoryStrategyType.MULTI_REVISION) + + when: + manager.download("v1.2") + then: + folder.resolve(REPOS_SUBDIR + '/nextflow-io/hello/' + REVISION_SUBDIR + '/1b420d060d3fad67027154ac48e3bdea06f058da/.git').isDirectory() + folder.resolve(REPOS_SUBDIR + '/nextflow-io/hello/' + BARE_REPO).isDirectory() + manager.getLocalPath().toString() == folder.resolve(REPOS_SUBDIR + '/nextflow-io/hello/' + REVISION_SUBDIR + '/1b420d060d3fad67027154ac48e3bdea06f058da').toString() + when: + manager.download("v1.2") + then: + noExceptionThrown() + + } + @Requires({System.getenv('NXF_GITHUB_ACCESS_TOKEN')}) - def testPullTagTwice() { + def 'test download from tag twice legacy'() { given: def folder = tempDir.getRoot() def token = System.getenv('NXF_GITHUB_ACCESS_TOKEN') def manager = new AssetManager().build('nextflow-io/hello', [providers: [github: [auth: token]]]) + manager.setStrategyType(AssetManager.RepositoryStrategyType.LEGACY) when: manager.download("v1.2") then: folder.resolve('nextflow-io/hello/.git').isDirectory() + manager.getLocalPath().toString() == folder.resolve('nextflow-io/hello').toString() when: manager.download("v1.2") @@ -197,14 +229,37 @@ class AssetManagerTest extends Specification { noExceptionThrown() } + @Requires({System.getenv('NXF_GITHUB_ACCESS_TOKEN')}) + def 'test download from hash twice with multi-revision'() { + + given: + def folder = tempDir.getRoot() + def token = System.getenv('NXF_GITHUB_ACCESS_TOKEN') + def manager = new AssetManager().build('nextflow-io/hello', [providers: [github: [auth: token]]]) + manager.setStrategyType(AssetManager.RepositoryStrategyType.MULTI_REVISION) + + when: + manager.download("6b9515aba6c7efc6a9b3f273ce116fc0c224bf68") + then: + folder.resolve(REPOS_SUBDIR + '/nextflow-io/hello/' + REVISION_SUBDIR + '/6b9515aba6c7efc6a9b3f273ce116fc0c224bf68/.git').isDirectory() + folder.resolve(REPOS_SUBDIR + '/nextflow-io/hello/' + BARE_REPO).isDirectory() + manager.getLocalPath().toString() == folder.resolve(REPOS_SUBDIR + '/nextflow-io/hello/' + REVISION_SUBDIR + '/6b9515aba6c7efc6a9b3f273ce116fc0c224bf68').toString() + + when: + def result = manager.download("6b9515aba6c7efc6a9b3f273ce116fc0c224bf68") + then: + noExceptionThrown() + } + // The hashes used here are NOT associated with tags. @Requires({System.getenv('NXF_GITHUB_ACCESS_TOKEN')}) - def testPullHashTwice() { + def 'test download from hash twice legacy'() { given: def folder = tempDir.getRoot() def token = System.getenv('NXF_GITHUB_ACCESS_TOKEN') def manager = new AssetManager().build('nextflow-io/hello', [providers: [github: [auth: token]]]) + manager.setStrategyType(AssetManager.RepositoryStrategyType.LEGACY) when: manager.download("6b9515aba6c7efc6a9b3f273ce116fc0c224bf68") @@ -222,12 +277,13 @@ class AssetManagerTest extends Specification { // Downloading a branch first and then pulling the branch // should work fine, unlike with tags. @Requires({System.getenv('NXF_GITHUB_ACCESS_TOKEN')}) - def testPullBranchTwice() { + def 'test download from branch twice legacy'() { given: def folder = tempDir.getRoot() def token = System.getenv('NXF_GITHUB_ACCESS_TOKEN') def manager = new AssetManager().build('nextflow-io/hello', [providers: [github: [auth: token]]]) + manager.setStrategyType(AssetManager.RepositoryStrategyType.LEGACY) when: manager.download("mybranch") @@ -240,17 +296,39 @@ class AssetManagerTest extends Specification { noExceptionThrown() } + @Requires({System.getenv('NXF_GITHUB_ACCESS_TOKEN')}) + def 'test download from branch twice with multi-revision'() { + + given: + def folder = tempDir.getRoot() + def token = System.getenv('NXF_GITHUB_ACCESS_TOKEN') + def manager = new AssetManager().build('nextflow-io/hello', [providers: [github: [auth: token]]]) + manager.setStrategyType(AssetManager.RepositoryStrategyType.MULTI_REVISION) + + when: + manager.download("mybranch") + then: + folder.resolve(REPOS_SUBDIR + '/nextflow-io/hello/' + BARE_REPO).isDirectory() + folder.resolve(REPOS_SUBDIR + '/nextflow-io/hello/' + REVISION_SUBDIR + '/1c3e9e7404127514d69369cd87f8036830f5cf64/.git').isDirectory() + manager.getLocalPath().toString() == folder.resolve(REPOS_SUBDIR + '/nextflow-io/hello/' + REVISION_SUBDIR + '/1c3e9e7404127514d69369cd87f8036830f5cf64').toString() + when: + manager.download("mybranch") + then: + noExceptionThrown() + } + // First clone a repo with a tag, then forget to include the -r argument // when you execute nextflow. // Note that while the download will work, execution will fail subsequently // at a separate check - this just tests that we don't fail because of a detached head. @Requires({System.getenv('NXF_GITHUB_ACCESS_TOKEN')}) - def testPullTagThenBranch() { + def 'test download tag then branch legacy'() { given: def folder = tempDir.getRoot() def token = System.getenv('NXF_GITHUB_ACCESS_TOKEN') def manager = new AssetManager().build('nextflow-io/hello', [providers: [github: [auth: token]]]) + manager.setStrategyType(AssetManager.RepositoryStrategyType.LEGACY) when: manager.download("v1.2") @@ -263,6 +341,31 @@ class AssetManagerTest extends Specification { noExceptionThrown() } + @Requires({System.getenv('NXF_GITHUB_ACCESS_TOKEN')}) + def 'test download tag then branch with multi-revision'() { + + given: + def folder = tempDir.getRoot() + def token = System.getenv('NXF_GITHUB_ACCESS_TOKEN') + def manager = new AssetManager().build('nextflow-io/hello', [providers: [github: [auth: token]]]) + manager.setStrategyType(AssetManager.RepositoryStrategyType.MULTI_REVISION) + + when: + manager.download("v1.2") + then: + folder.resolve(REPOS_SUBDIR + '/nextflow-io/hello/' + REVISION_SUBDIR + '/1b420d060d3fad67027154ac48e3bdea06f058da/.git').isDirectory() + folder.resolve(REPOS_SUBDIR + '/nextflow-io/hello/' + BARE_REPO).isDirectory() + manager.getLocalPath().toString() == folder.resolve(REPOS_SUBDIR + '/nextflow-io/hello/' + REVISION_SUBDIR + '/1b420d060d3fad67027154ac48e3bdea06f058da').toString() + + when: + manager.download("mybranch") + then: + folder.resolve(REPOS_SUBDIR + '/nextflow-io/hello/' + BARE_REPO).isDirectory() + folder.resolve(REPOS_SUBDIR + '/nextflow-io/hello/' + REVISION_SUBDIR + '/1c3e9e7404127514d69369cd87f8036830f5cf64/.git').isDirectory() + manager.getLocalPath().toString() == folder.resolve(REPOS_SUBDIR + '/nextflow-io/hello/' + REVISION_SUBDIR + '/1c3e9e7404127514d69369cd87f8036830f5cf64').toString() + noExceptionThrown() + } + @Requires({System.getenv('NXF_GITHUB_ACCESS_TOKEN')}) def testClone() { @@ -447,7 +550,7 @@ class AssetManagerTest extends Specification { } - def 'should create a script file object' () { + def 'should create a script file object legacy' () { given: def dir = tempDir.root @@ -546,12 +649,13 @@ class AssetManagerTest extends Specification { } @Requires({System.getenv('NXF_GITHUB_ACCESS_TOKEN')}) - def 'should download branch specified'() { + def 'should download branch specified legacy'() { given: def folder = tempDir.getRoot() def token = System.getenv('NXF_GITHUB_ACCESS_TOKEN') def manager = new AssetManager().build('nextflow-io/nf-test-branch', [providers: [github: [auth: token]]]) + manager.setStrategyType(AssetManager.RepositoryStrategyType.LEGACY) when: manager.download("dev") @@ -566,6 +670,28 @@ class AssetManagerTest extends Specification { noExceptionThrown() } + @Requires({System.getenv('NXF_GITHUB_ACCESS_TOKEN')}) + def 'should download branch specified multi-revision'() { + + given: + def folder = tempDir.getRoot() + def token = System.getenv('NXF_GITHUB_ACCESS_TOKEN') + def manager = new AssetManager().build('nextflow-io/nf-test-branch', [providers: [github: [auth: token]]]) + manager.setStrategyType(AssetManager.RepositoryStrategyType.MULTI_REVISION) + + when: + manager.download("dev") + then: + folder.resolve(REPOS_SUBDIR + '/nextflow-io/nf-test-branch/' + REVISION_SUBDIR + '/6f882561d589365c3950d170df8445e3c0dc8028/.git').isDirectory() + and: + folder.resolve(REPOS_SUBDIR + '/nextflow-io/nf-test-branch/' + REVISION_SUBDIR + '/6f882561d589365c3950d170df8445e3c0dc8028/workflow.nf').text == "println 'Hello'\n" + + when: + manager.download() + then: + noExceptionThrown() + } + @Requires({System.getenv('NXF_GITHUB_ACCESS_TOKEN')}) def 'should fetch main script from branch specified'() { @@ -581,12 +707,13 @@ class AssetManagerTest extends Specification { } @Requires({System.getenv('NXF_GITHUB_ACCESS_TOKEN')}) - def 'should download tag specified'() { + def 'should download tag specified legacy'() { given: def folder = tempDir.getRoot() def token = System.getenv('NXF_GITHUB_ACCESS_TOKEN') def manager = new AssetManager().build('nextflow-io/nf-test-branch', [providers: [github: [auth: token]]]) + manager.setStrategyType(AssetManager.RepositoryStrategyType.LEGACY) when: manager.download("v0.1") @@ -595,6 +722,27 @@ class AssetManagerTest extends Specification { and: folder.resolve('nextflow-io/nf-test-branch/workflow.nf').text == "println 'Hello'\n" + when: + manager.download() + then: + noExceptionThrown() + } + @Requires({System.getenv('NXF_GITHUB_ACCESS_TOKEN')}) + def 'should download tag specified with revision'() { + + given: + def folder = tempDir.getRoot() + def token = System.getenv('NXF_GITHUB_ACCESS_TOKEN') + def manager = new AssetManager().build('nextflow-io/nf-test-branch', [providers: [github: [auth: token]]]) + manager.setStrategyType(AssetManager.RepositoryStrategyType.MULTI_REVISION) + + when: + manager.download("v0.1") + then: + folder.resolve(REPOS_SUBDIR + '/nextflow-io/nf-test-branch/' + REVISION_SUBDIR + '/6f882561d589365c3950d170df8445e3c0dc8028/.git').isDirectory() + and: + folder.resolve(REPOS_SUBDIR + '/nextflow-io/nf-test-branch/' + REVISION_SUBDIR + '/6f882561d589365c3950d170df8445e3c0dc8028/workflow.nf').text == "println 'Hello'\n" + when: manager.download() then: @@ -602,19 +750,17 @@ class AssetManagerTest extends Specification { } @Requires({System.getenv('NXF_GITHUB_ACCESS_TOKEN')}) - def 'should identify default branch when downloading repo'() { + def 'should identify default branch when downloading repo legacy'() { given: - def folder = tempDir.getRoot() def token = System.getenv('NXF_GITHUB_ACCESS_TOKEN') def manager = new AssetManager().build('nextflow-io/socks', [providers: [github: [auth: token]]]) - + manager.setStrategyType(AssetManager.RepositoryStrategyType.LEGACY) when: // simulate calling `nextflow run nextflow-io/socks` without specifying a revision manager.download() manager.checkout(null) then: - folder.resolve('nextflow-io/socks/.git').isDirectory() manager.getCurrentRevision() == 'main' when: @@ -624,31 +770,422 @@ class AssetManagerTest extends Specification { } @Requires({System.getenv('NXF_GITHUB_ACCESS_TOKEN')}) - def 'can filter remote branches'() { + def 'should identify default branch when downloading repo multi-revision'() { + given: - def folder = tempDir.getRoot() def token = System.getenv('NXF_GITHUB_ACCESS_TOKEN') - def manager = new AssetManager().build('nextflow-io/hello', [providers: [github: [auth: token]]]) + def manager = new AssetManager().build('nextflow-io/socks', [providers: [github: [auth: token]]]) + + when: + // simulate calling `nextflow run nextflow-io/socks` without specifying a revision + manager.download() + manager.checkout(null) + then: + manager.getCurrentRevision() == 'main' + + when: manager.download() - def branches = manager.getBranchList() + then: + noExceptionThrown() + } + + @Requires({System.getenv('NXF_GITHUB_ACCESS_TOKEN')}) + def 'should list revisions and commits'() { + given: + def token = System.getenv('NXF_GITHUB_ACCESS_TOKEN') + def pipelineName ='nextflow-io/hello' + def revision1 = 'v1.2' + def revision2 = 'mybranch' + def manager = new AssetManager().build(pipelineName, [providers: [github: [auth: token]]]) + manager.download(revision1) + manager.download(revision2) + + when: + def branchesAndTags = manager.getBranchesAndTags(false) + then: + def pulled = branchesAndTags.pulled as List + pulled.size() == 2 + revision1 in pulled && revision2 in pulled + def branches = branchesAndTags.branches as List + def tags = branchesAndTags.tags as List + tags.find {it.name == revision1 }.commitId == '0ec2ecd0ac13bc7e32594c0258ebce55e383d241' + branches.find { it.name == revision2 }.commitId == '1c3e9e7404127514d69369cd87f8036830f5cf64' + } + + def 'should select multi-revision strategy for uninitialized repository'() { + given: + def folder = tempDir.getRoot() + // No repositories exist yet when: - def remote_head = branches.find { it.name == 'refs/remotes/origin/HEAD' } + def manager = new AssetManager().build('test-org/test-repo') + then: - remote_head != null - !AssetManager.isRemoteBranch(remote_head) + manager.isNotInitialized() + manager.isUsingMultiRevisionStrategy() + !manager.isUsingLegacyStrategy() + } + + def 'should select legacy strategy when legacy repository exists'() { + given: + def folder = tempDir.getRoot() + def legacyPath = folder.resolve('test-org/test-repo') + legacyPath.mkdirs() + + // Create a proper git repository + def init = Git.init() + def repo = init.setDirectory(legacyPath.toFile()).call() + repo.close() + + // Add git config with remote url + legacyPath.resolve('.git/config').text = GIT_CONFIG_TEXT when: - def remote_master = branches.find { it.name == 'refs/remotes/origin/master' } + def manager = new AssetManager().build('test-org/test-repo') + then: - remote_master != null - AssetManager.isRemoteBranch(remote_master) + manager.isOnlyLegacy() + manager.isUsingLegacyStrategy() + !manager.isUsingMultiRevisionStrategy() + } + + def 'should select multi-revision strategy when bare repository exists'() { + given: + def folder = tempDir.getRoot() + def barePath = folder.resolve(REPOS_SUBDIR + '/test-org/test-repo/' + BARE_REPO) + barePath.mkdirs() + + // Create a proper bare git repository + def init = Git.init() + init.setDirectory(barePath.toFile()) + init.setBare(true) + def repo = init.call() + repo.close() + + when: + def manager = new AssetManager().build('test-org/test-repo') + + then: + manager.isUsingMultiRevisionStrategy() + !manager.isUsingLegacyStrategy() + } + + def 'should prefer multi-revision strategy for hybrid repository'() { + given: + def folder = tempDir.getRoot() + // Create legacy repository + def legacyPath = folder.resolve('test-org/test-repo') + legacyPath.mkdirs() + def initLegacy = Git.init() + def repoLegacy = initLegacy.setDirectory(legacyPath.toFile()).call() + repoLegacy.close() + legacyPath.resolve('.git/config').text = GIT_CONFIG_TEXT + + // Create bare repository + def barePath = folder.resolve(REPOS_SUBDIR + '/test-org/test-repo/' + BARE_REPO) + barePath.mkdirs() + def initBare = Git.init() + initBare.setDirectory(barePath.toFile()) + initBare.setBare(true) + def repoBare = initBare.call() + repoBare.close() + barePath.resolve('config').text = GIT_CONFIG_TEXT when: - def local_master = branches.find { it.name == 'refs/heads/master' } + def manager = new AssetManager().build('test-org/test-repo') + then: - local_master != null - !AssetManager.isRemoteBranch(local_master) + manager.isUsingMultiRevisionStrategy() + !manager.isUsingLegacyStrategy() } + def 'should force legacy strategy when NXF_SCM_LEGACY is set'() { + given: + def folder = tempDir.getRoot() + // No repositories exist yet + and: + def originalValue = System.getenv('NXF_SCM_LEGACY') + + when: + // Simulate NXF_SCM_LEGACY being set to true + nextflow.SysEnv.push([NXF_SCM_LEGACY: 'true']) + def manager = new AssetManager().build('test-org/test-repo') + + then: + manager.isNotInitialized() + manager.isUsingLegacyStrategy() + !manager.isUsingMultiRevisionStrategy() + + cleanup: + nextflow.SysEnv.pop() + } + + def 'should switch strategy when explicitly set'() { + given: + def folder = tempDir.getRoot() + + when: + def manager = new AssetManager().build('test-org/test-repo') + then: + manager.isUsingMultiRevisionStrategy() + + when: + manager.setStrategyType(AssetManager.RepositoryStrategyType.LEGACY) + then: + manager.isUsingLegacyStrategy() + + when: + manager.setStrategyType(AssetManager.RepositoryStrategyType.MULTI_REVISION) + then: + manager.isUsingMultiRevisionStrategy() + } + + def 'should not switch strategy if already using requested type'() { + given: + def folder = tempDir.getRoot() + def manager = new AssetManager().build('test-org/test-repo') + + when: + def strategyBefore = manager.@strategy + manager.setStrategyType(AssetManager.RepositoryStrategyType.MULTI_REVISION) + def strategyAfter = manager.@strategy + + then: + strategyBefore.is(strategyAfter) + manager.isUsingMultiRevisionStrategy() + } + + def 'should detect repository status correctly'() { + given: + def folder = tempDir.getRoot() + + expect: 'no repository exists' + new AssetManager().build('test-org/test-repo').isNotInitialized() + + + when: 'only legacy exists' + def legacyPath2 = folder.resolve('test-org/repo2') + legacyPath2.mkdirs() + legacyPath2.resolve('.git').mkdir() + legacyPath2.resolve('.git/config').text = GIT_CONFIG_TEXT + def manager = new AssetManager().build('test-org/repo2') + then: + !manager.isNotInitialized() + manager.isOnlyLegacy() + + when: 'only bare exists' + def barePath3 = folder.resolve(REPOS_SUBDIR + '/test-org/repo3/' + BARE_REPO) + barePath3.mkdirs() + def initBare3 = Git.init() + initBare3.setDirectory(barePath3.toFile()) + initBare3.setBare(true) + def repoBare3 = initBare3.call() + repoBare3.close() + barePath3.resolve('config').text = GIT_CONFIG_TEXT + manager = new AssetManager().build('test-org/repo3') + then: + !manager.isNotInitialized() + !manager.isOnlyLegacy() + MultiRevisionRepositoryStrategy.checkProject(folder.toFile(), 'test-org/repo3') + + when: 'both exist' + def legacyPath4 = folder.resolve('test-org/repo4') + legacyPath4.mkdirs() + legacyPath4.resolve('.git').mkdir() + legacyPath4.resolve('.git/config').text = GIT_CONFIG_TEXT + def barePath4 = folder.resolve(REPOS_SUBDIR + '/test-org/repo4/' + BARE_REPO) + barePath4.mkdirs() + def initBare4 = Git.init() + initBare4.setDirectory(barePath4.toFile()) + initBare4.setBare(true) + def repoBare4 = initBare4.call() + repoBare4.close() + barePath4.resolve('config').text = GIT_CONFIG_TEXT + manager = new AssetManager().build('test-org/repo4') + then: + !manager.isNotInitialized() + !manager.isOnlyLegacy() + LegacyRepositoryStrategy.checkProject(folder.toFile(), 'test-org/repo4') + MultiRevisionRepositoryStrategy.checkProject(folder.toFile(), 'test-org/repo4') + } + + // ============================================ + // DROP OPERATIONS TESTS + // ============================================ + + @Requires({System.getenv('NXF_GITHUB_ACCESS_TOKEN')}) + def 'should drop a specific revision with multi-revision strategy'() { + given: + def token = System.getenv('NXF_GITHUB_ACCESS_TOKEN') + def folder = tempDir.getRoot() + def pipelineName = 'nextflow-io/hello' + def revision1 = 'v1.2' + def revision2 = 'mybranch' + + and: 'download two revisions' + def manager = new AssetManager().build(pipelineName, [providers: [github: [auth: token]]]) + manager.setStrategyType(AssetManager.RepositoryStrategyType.MULTI_REVISION) + manager.download(revision1) + manager.download(revision2) + + and: 'verify both revisions exist' + def commit1Path = folder.resolve(REPOS_SUBDIR + '/nextflow-io/hello/' + REVISION_SUBDIR + '/1b420d060d3fad67027154ac48e3bdea06f058da') + def commit2Path = folder.resolve(REPOS_SUBDIR + '/nextflow-io/hello/' + REVISION_SUBDIR + '/1c3e9e7404127514d69369cd87f8036830f5cf64') + commit1Path.exists() + commit2Path.exists() + + when: 'drop the first revision' + manager.drop(revision1) + + then: 'first revision is deleted but second remains' + !commit1Path.exists() + commit2Path.exists() + folder.resolve(REPOS_SUBDIR + '/nextflow-io/hello/' + BARE_REPO).exists() + } + + @Requires({System.getenv('NXF_GITHUB_ACCESS_TOKEN')}) + def 'should drop all revisions with multi-revision strategy'() { + given: + def token = System.getenv('NXF_GITHUB_ACCESS_TOKEN') + def folder = tempDir.getRoot() + def pipelineName = 'nextflow-io/hello' + def revision1 = 'v1.2' + def revision2 = 'mybranch' + + and: 'download two revisions' + def manager = new AssetManager().build(pipelineName, [providers: [github: [auth: token]]]) + manager.setStrategyType(AssetManager.RepositoryStrategyType.MULTI_REVISION) + manager.download(revision1) + manager.download(revision2) + + and: 'verify both revisions and bare repo exist' + def projectPath = folder.resolve(REPOS_SUBDIR + '/nextflow-io/hello') + def commit1Path = projectPath.resolve(REVISION_SUBDIR + '/1b420d060d3fad67027154ac48e3bdea06f058da') + def commit2Path = projectPath.resolve(REVISION_SUBDIR + '/1c3e9e7404127514d69369cd87f8036830f5cf64') + def barePath = projectPath.resolve(BARE_REPO) + commit1Path.exists() + commit2Path.exists() + barePath.exists() + + when: 'drop all revisions (no revision specified)' + manager.drop(null) + + then: 'everything is deleted including bare repo' + !commit1Path.exists() + !commit2Path.exists() + !barePath.exists() + !projectPath.exists() + } + + @Requires({System.getenv('NXF_GITHUB_ACCESS_TOKEN')}) + def 'should not drop revision with uncommitted changes unless forced'() { + given: + def token = System.getenv('NXF_GITHUB_ACCESS_TOKEN') + def folder = tempDir.getRoot() + def pipelineName = 'nextflow-io/hello' + def revision = 'v1.2' + + and: 'download a revision' + def manager = new AssetManager().build(pipelineName, [providers: [github: [auth: token]]]) + manager.setStrategyType(AssetManager.RepositoryStrategyType.MULTI_REVISION) + manager.download(revision) + + and: 'make local changes' + def commitPath = folder.resolve(REPOS_SUBDIR + '/nextflow-io/hello/' + REVISION_SUBDIR + '/1b420d060d3fad67027154ac48e3bdea06f058da') + commitPath.resolve('test-file.txt').text = 'uncommitted change' + + when: 'try to drop without force' + manager.drop(revision, false) + + then: 'operation fails' + thrown(AbortOperationException) + commitPath.exists() + + when: 'drop with force flag' + manager.drop(revision, true) + + then: 'revision is deleted' + !commitPath.exists() + } + + @Requires({System.getenv('NXF_GITHUB_ACCESS_TOKEN')}) + def 'should handle drop of non-existent revision gracefully'() { + given: + def token = System.getenv('NXF_GITHUB_ACCESS_TOKEN') + def pipelineName = 'nextflow-io/hello' + + and: 'create manager downloading a revision' + def manager = new AssetManager().build(pipelineName, [providers: [github: [auth: token]]]) + manager.setStrategyType(AssetManager.RepositoryStrategyType.MULTI_REVISION) + manager.download("v1.2") + + when: 'try to drop a revision that was never downloaded' + manager.drop('nonexistent-revision') + + then: 'no exception is thrown' + noExceptionThrown() + } + + // ============================================ + // REVISION SWITCHING TESTS + // ============================================ + + @Requires({System.getenv('NXF_GITHUB_ACCESS_TOKEN')}) + def 'should switch between downloaded revisions'() { + given: + def token = System.getenv('NXF_GITHUB_ACCESS_TOKEN') + def folder = tempDir.getRoot() + def pipelineName = 'nextflow-io/hello' + def revision1 = 'v1.2' + def revision2 = 'mybranch' + + and: 'download two revisions' + def manager = new AssetManager().build(pipelineName, [providers: [github: [auth: token]]]) + manager.setStrategyType(AssetManager.RepositoryStrategyType.MULTI_REVISION) + manager.download(revision1) + manager.download(revision2) + + when: 'checkout first revision' + manager.checkout(revision1) + + then: 'local path points to first revision' + manager.getLocalPath().toString().contains('1b420d060d3fad67027154ac48e3bdea06f058da') + manager.getCurrentRevision() == revision1 + + when: 'checkout second revision' + manager.checkout(revision2) + + then: 'local path points to second revision' + manager.getLocalPath().toString().contains('1c3e9e7404127514d69369cd87f8036830f5cf64') + manager.getCurrentRevision() == revision2 + } + + @Requires({System.getenv('NXF_GITHUB_ACCESS_TOKEN')}) + def 'should use setRevision to switch between commits'() { + given: + def token = System.getenv('NXF_GITHUB_ACCESS_TOKEN') + def pipelineName = 'nextflow-io/hello' + def revision1 = 'v1.2' + def revision2 = 'mybranch' + + and: 'create manager and download revisions' + def manager = new AssetManager().build(pipelineName, [providers: [github: [auth: token]]]) + manager.setStrategyType(AssetManager.RepositoryStrategyType.MULTI_REVISION) + manager.download(revision1) + manager.download(revision2) + + when: 'use setRevision' + manager.setRevision(revision1) + + then: 'revision is updated' + manager.getLocalPath().toString().contains('1b420d060d3fad67027154ac48e3bdea06f058da') + + when: 'switch to another revision' + manager.setRevision(revision2) + + then: 'local path updated' + manager.getLocalPath().toString().contains('1c3e9e7404127514d69369cd87f8036830f5cf64') + } + + } diff --git a/modules/nextflow/src/test/groovy/nextflow/scm/GitReferenceHelperTest.groovy b/modules/nextflow/src/test/groovy/nextflow/scm/GitReferenceHelperTest.groovy new file mode 100644 index 0000000000..6d8a7637ef --- /dev/null +++ b/modules/nextflow/src/test/groovy/nextflow/scm/GitReferenceHelperTest.groovy @@ -0,0 +1,324 @@ +/* + * Copyright 2013-2025, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package nextflow.scm + +import org.eclipse.jgit.lib.ObjectId +import org.eclipse.jgit.lib.Ref +import org.junit.Rule +import spock.lang.Requires +import spock.lang.Specification +import test.TemporaryPath + +class GitReferenceHelperTest extends Specification { + + @Rule + TemporaryPath tempDir = new TemporaryPath() + + def setup() { + AssetManager.root = tempDir.root.toFile() + } + + @Requires({System.getenv('NXF_GITHUB_ACCESS_TOKEN')}) + def 'can filter remote branches'() { + given: + def folder = tempDir.getRoot() + def token = System.getenv('NXF_GITHUB_ACCESS_TOKEN') + def manager = new AssetManager().build('nextflow-io/hello', [providers: [github: [auth: token]]]) + manager.setStrategyType(AssetManager.RepositoryStrategyType.LEGACY) + manager.download() + def branches = manager.getBranchList() + + when: + def remote_head = branches.find { it.name == 'refs/remotes/origin/HEAD' } + then: + remote_head != null + !GitReferenceHelper.isRemoteBranch(remote_head) + + when: + def remote_master = branches.find { it.name == 'refs/remotes/origin/master' } + then: + remote_master != null + GitReferenceHelper.isRemoteBranch(remote_master) + + when: + def local_master = branches.find { it.name == 'refs/heads/master' } + then: + local_master != null + !GitReferenceHelper.isRemoteBranch(local_master) + } + + def 'should check hasRemoteChange with no remote'() { + given: + def ref = Mock(Ref) + + expect: + !GitReferenceHelper.hasRemoteChange(ref, null) + !GitReferenceHelper.hasRemoteChange(ref, [:]) + } + + def 'should check hasRemoteChange when ref not in remote'() { + given: + def ref = Mock(Ref) { + getName() >> 'refs/heads/master' + } + def remote = ['refs/heads/other': Mock(Ref)] + + expect: + !GitReferenceHelper.hasRemoteChange(ref, remote) + } + + def 'should detect hasRemoteChange when ObjectIds differ'() { + given: + def localId = ObjectId.fromString('1234567890123456789012345678901234567890') + def remoteId = ObjectId.fromString('abcdefabcdefabcdefabcdefabcdefabcdefabcd') + + def localRef = Mock(Ref) { + getName() >> 'refs/heads/master' + getObjectId() >> localId + } + def remoteRef = Mock(Ref) { + getObjectId() >> remoteId + } + def remote = ['refs/heads/master': remoteRef] + + expect: + GitReferenceHelper.hasRemoteChange(localRef, remote) + } + + def 'should not detect hasRemoteChange when ObjectIds are same'() { + given: + def sameId = ObjectId.fromString('1234567890123456789012345678901234567890') + + def localRef = Mock(Ref) { + getName() >> 'refs/heads/master' + getObjectId() >> sameId + } + def remoteRef = Mock(Ref) { + getObjectId() >> sameId + } + def remote = ['refs/heads/master': remoteRef] + + expect: + !GitReferenceHelper.hasRemoteChange(localRef, remote) + } + + def 'should format ObjectId in human readable form'() { + given: + def objectId = ObjectId.fromString('1234567890123456789012345678901234567890') + + when: + def result = GitReferenceHelper.formatObjectId(objectId, true) + + then: + result == '1234567890' + } + + def 'should format ObjectId in full form'() { + given: + def objectId = ObjectId.fromString('1234567890123456789012345678901234567890') + + when: + def result = GitReferenceHelper.formatObjectId(objectId, false) + + then: + result == '1234567890123456789012345678901234567890' + } + + def 'should format update with level 0'() { + given: + def objectId = ObjectId.fromString('1234567890123456789012345678901234567890') + def remoteRef = Mock(Ref) { + getObjectId() >> objectId + } + + when: + def result = GitReferenceHelper.formatUpdate(remoteRef, 0) + + then: + result == 'updates on remote' + } + + def 'should format update with level 1'() { + given: + def objectId = ObjectId.fromString('1234567890123456789012345678901234567890') + def remoteRef = Mock(Ref) { + getObjectId() >> objectId + } + + when: + def result = GitReferenceHelper.formatUpdate(remoteRef, 1) + + then: + result == 'updates on remote 1234567890' + } + + def 'should format update with level 2'() { + given: + def objectId = ObjectId.fromString('1234567890123456789012345678901234567890') + def remoteRef = Mock(Ref) { + getObjectId() >> objectId + } + + when: + def result = GitReferenceHelper.formatUpdate(remoteRef, 2) + + then: + result == 'updates on remote 1234567890123456789012345678901234567890' + } + + def 'should shorten remote ref name'() { + expect: + GitReferenceHelper.shortenRefName('refs/remotes/origin/master') == 'master' + GitReferenceHelper.shortenRefName('refs/remotes/origin/develop') == 'develop' + GitReferenceHelper.shortenRefName('refs/remotes/origin/feature/test') == 'feature/test' + } + + def 'should shorten standard ref names'() { + expect: + GitReferenceHelper.shortenRefName('refs/heads/master') == 'master' + GitReferenceHelper.shortenRefName('refs/tags/v1.0.0') == 'v1.0.0' + } + + def 'should not shorten already short ref names'() { + expect: + GitReferenceHelper.shortenRefName('master') == 'master' + GitReferenceHelper.shortenRefName('develop') == 'develop' + } + + def 'should convert ref to map without remote'() { + given: + def objectId = ObjectId.fromString('1234567890123456789012345678901234567890') + def ref = Mock(Ref) { + getName() >> 'refs/heads/master' + getObjectId() >> objectId + getPeeledObjectId() >> null + } + + when: + def result = GitReferenceHelper.refToMap(ref, null) + + then: + result.name == 'master' + result.commitId == '1234567890123456789012345678901234567890' + !result.containsKey('latestId') + } + + def 'should convert ref to map with peeled ObjectId'() { + given: + def objectId = ObjectId.fromString('1234567890123456789012345678901234567890') + def peeledId = ObjectId.fromString('abcdefabcdefabcdefabcdefabcdefabcdefabcd') + def ref = Mock(Ref) { + getName() >> 'refs/tags/v1.0.0' + getObjectId() >> objectId + getPeeledObjectId() >> peeledId + } + + when: + def result = GitReferenceHelper.refToMap(ref, null) + + then: + result.name == 'v1.0.0' + result.commitId == 'abcdefabcdefabcdefabcdefabcdefabcdefabcd' + !result.containsKey('latestId') + } + + def 'should convert ref to map with remote changes'() { + given: + def localId = ObjectId.fromString('1234567890123456789012345678901234567890') + def remoteId = ObjectId.fromString('abcdefabcdefabcdefabcdefabcdefabcdefabcd') + + def ref = Mock(Ref) { + getName() >> 'refs/heads/master' + getObjectId() >> localId + getPeeledObjectId() >> null + } + def remoteRef = Mock(Ref) { + getObjectId() >> remoteId + } + def remote = ['refs/heads/master': remoteRef] + + when: + def result = GitReferenceHelper.refToMap(ref, remote) + + then: + result.name == 'master' + result.commitId == '1234567890123456789012345678901234567890' + result.latestId == 'abcdefabcdefabcdefabcdefabcdefabcdefabcd' + } + + def 'should convert ref to map without remote changes'() { + given: + def sameId = ObjectId.fromString('1234567890123456789012345678901234567890') + + def ref = Mock(Ref) { + getName() >> 'refs/heads/master' + getObjectId() >> sameId + getPeeledObjectId() >> null + } + def remoteRef = Mock(Ref) { + getObjectId() >> sameId + } + def remote = ['refs/heads/master': remoteRef] + + when: + def result = GitReferenceHelper.refToMap(ref, remote) + + then: + result.name == 'master' + result.commitId == '1234567890123456789012345678901234567890' + !result.containsKey('latestId') + } + + def 'should find ref in commits using ObjectId'() { + given: + def objectId = ObjectId.fromString('1234567890123456789012345678901234567890') + def ref = Mock(Ref) { + getObjectId() >> objectId + getPeeledObjectId() >> null + } + def commits = ['abcdefabcdefabcdefabcdefabcdefabcdefabcd', '1234567890123456789012345678901234567890'] + + expect: + GitReferenceHelper.isRefInCommits(ref, commits) + } + + def 'should find ref in commits using peeled ObjectId'() { + given: + def objectId = ObjectId.fromString('1234567890123456789012345678901234567890') + def peeledId = ObjectId.fromString('abcdefabcdefabcdefabcdefabcdefabcdefabcd') + def ref = Mock(Ref) { + getObjectId() >> objectId + getPeeledObjectId() >> peeledId + } + def commits = ['abcdefabcdefabcdefabcdefabcdefabcdefabcd', 'fedcbafedcbafedcbafedcbafedcbafedcbafed'] + + expect: + GitReferenceHelper.isRefInCommits(ref, commits) + } + + def 'should not find ref in commits when not present'() { + given: + def objectId = ObjectId.fromString('1234567890123456789012345678901234567890') + def ref = Mock(Ref) { + getObjectId() >> objectId + getPeeledObjectId() >> null + } + def commits = ['abcdefabcdefabcdefabcdefabcdefabcdefabcd', 'fedcbafedcbafedcbafedcbafedcbafedcbafed'] + + expect: + !GitReferenceHelper.isRefInCommits(ref, commits) + } +} diff --git a/modules/nextflow/src/test/groovy/nextflow/scm/MultiRevisionRepositoryStrategyTest.groovy b/modules/nextflow/src/test/groovy/nextflow/scm/MultiRevisionRepositoryStrategyTest.groovy new file mode 100644 index 0000000000..d5971ec4de --- /dev/null +++ b/modules/nextflow/src/test/groovy/nextflow/scm/MultiRevisionRepositoryStrategyTest.groovy @@ -0,0 +1,130 @@ +/* + * Copyright 2013-2025, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package nextflow.scm + +import nextflow.config.Manifest + +import static MultiRevisionRepositoryStrategy.BARE_REPO +import static MultiRevisionRepositoryStrategy.REVISION_SUBDIR +import static nextflow.scm.MultiRevisionRepositoryStrategy.REPOS_SUBDIR + +import spock.lang.IgnoreIf + +import org.junit.Rule +import spock.lang.Requires +import spock.lang.Specification +import test.TemporaryPath + +/** + * + * @author Jorge Ejarque + */ +class MultiRevisionRepositoryStrategyTest extends Specification { + + @Rule + TemporaryPath tempDir = new TemporaryPath() + + def setup() { + AssetManager.root = tempDir.root.toFile() + } + + private MultiRevisionRepositoryStrategy createStrategy(String project, String token) { + final strategy = new MultiRevisionRepositoryStrategy(project) + if( token ) + strategy.setProvider(new GithubRepositoryProvider(project, new ProviderConfig('github').setAuth(token))) + return strategy + } + + def 'should list commits'() { + given: + def folder = tempDir.getRoot() + + when: + def strategy = createStrategy('cbcrg/pipe1', null) + folder.resolve(REPOS_SUBDIR + '/cbcrg/pipe1/' + REVISION_SUBDIR + '/12345').mkdirs() + folder.resolve(REPOS_SUBDIR + '/cbcrg/pipe1/' + REVISION_SUBDIR + '/67890').mkdirs() + def list = strategy.listDownloadedCommits() + then: + list.sort() == ['12345', '67890'] + + when: + strategy = createStrategy('cbcrg/pipe2', null) + folder.resolve(REPOS_SUBDIR + '/cbcrg/pipe2/' + REVISION_SUBDIR + '/abcde').mkdirs() + folder.resolve(REPOS_SUBDIR + '/cbcrg/pipe2/' + REVISION_SUBDIR + '/fghij').mkdirs() + list = strategy.listDownloadedCommits() + then: + list.sort() == ['abcde', 'fghij'] + } + + @IgnoreIf({ System.getenv('NXF_SMOKE') }) + @Requires({ System.getenv('NXF_GITHUB_ACCESS_TOKEN') }) + def 'should clone bare repo and get revisions'() { + + given: + def folder = tempDir.getRoot() + def token = System.getenv('NXF_GITHUB_ACCESS_TOKEN') + def strategy = createStrategy('nextflow-io/hello', token) + def manifest = Mock(Manifest) { + getDefaultBranch() >> 'master' + getRecurseSubmodules() >> false + } + + when: + strategy.checkBareRepo(manifest) + then: + folder.resolve(REPOS_SUBDIR + '/nextflow-io/hello/' + BARE_REPO).isDirectory() + folder.resolve(REPOS_SUBDIR + '/nextflow-io/hello/' + BARE_REPO + '/config').exists() + + expect: + strategy.revisionToCommitWithBareRepo('v1.2') == '1b420d060d3fad67027154ac48e3bdea06f058da' + + } + + @IgnoreIf({ System.getenv('NXF_SMOKE') }) + @Requires({ System.getenv('NXF_GITHUB_ACCESS_TOKEN') }) + def 'should create shared clone from commit, tag and branch'() { + + given: + def folder = tempDir.getRoot() + def token = System.getenv('NXF_GITHUB_ACCESS_TOKEN') + def strategy = createStrategy('nextflow-io/hello', token) + def manifest = Mock(Manifest) { + getDefaultBranch() >> 'master' + getRecurseSubmodules() >> false + } + + when: + strategy.download('7588c46ffefb4e3c06d4ab32c745c4d5e56cdad8', 1, manifest) + then: + folder.resolve(REPOS_SUBDIR + '/nextflow-io/hello/' + REVISION_SUBDIR + '/7588c46ffefb4e3c06d4ab32c745c4d5e56cdad8/.git').isDirectory() + folder.resolve(REPOS_SUBDIR + '/nextflow-io/hello/' + REVISION_SUBDIR + '/7588c46ffefb4e3c06d4ab32c745c4d5e56cdad8/.git/objects/info/alternates').text == folder.resolve(REPOS_SUBDIR + '/nextflow-io/hello/' + BARE_REPO + '/objects').toAbsolutePath().toString() + + when: + // tag v1.2 -> commit 1b420d060d3fad67027154ac48e3bdea06f058da + strategy.download('v1.2', 1, manifest) + then: + folder.resolve(REPOS_SUBDIR + '/nextflow-io/hello/' + REVISION_SUBDIR + '/1b420d060d3fad67027154ac48e3bdea06f058da/.git').isDirectory() + folder.resolve(REPOS_SUBDIR + '/nextflow-io/hello/' + REVISION_SUBDIR + '/1b420d060d3fad67027154ac48e3bdea06f058da/.git/objects/info/alternates').text == folder.resolve(REPOS_SUBDIR + '/nextflow-io/hello/' + BARE_REPO + '/objects').toAbsolutePath().toString() + + when: + strategy.download('mybranch', 1, manifest) + then: + folder.resolve(REPOS_SUBDIR + '/nextflow-io/hello/' + REVISION_SUBDIR + '/1c3e9e7404127514d69369cd87f8036830f5cf64/.git').isDirectory() + folder.resolve(REPOS_SUBDIR + '/nextflow-io/hello/' + REVISION_SUBDIR + '/1c3e9e7404127514d69369cd87f8036830f5cf64/.git/objects/info/alternates').text == folder.resolve(REPOS_SUBDIR + '/nextflow-io/hello/' + BARE_REPO + '/objects').toAbsolutePath().toString() + } + +} diff --git a/modules/nextflow/src/test/groovy/nextflow/scm/UpdateModuleTest.groovy b/modules/nextflow/src/test/groovy/nextflow/scm/UpdateModuleTest.groovy index d3e82abf72..f846b02fa6 100644 --- a/modules/nextflow/src/test/groovy/nextflow/scm/UpdateModuleTest.groovy +++ b/modules/nextflow/src/test/groovy/nextflow/scm/UpdateModuleTest.groovy @@ -101,8 +101,9 @@ class UpdateModuleTest extends Specification { def target = baseFolder.resolve('target') AssetManager.root = target.toFile() - when: + when: 'legacy' def manager = new AssetManager("file:${baseFolder}/pipe_x") + manager.setStrategyType(AssetManager.RepositoryStrategyType.LEGACY) manager.download() manager.updateModules() @@ -118,6 +119,25 @@ class UpdateModuleTest extends Specification { target.resolve('local/pipe_x/prj_bbb').exists() target.resolve('local/pipe_x/prj_bbb/file1.txt').text == 'Ciao' target.resolve('local/pipe_x/prj_bbb/file2.log').text == 'Mondo' + + when: 'multi-revision' + manager = new AssetManager("file:${baseFolder}/pipe_x") + manager.setStrategyType(AssetManager.RepositoryStrategyType.MULTI_REVISION) + manager.download() + manager.updateModules() + + then: + def local = manager.getLocalPath().toPath() + local.resolve('.git').exists() + local.resolve('main.nf').exists() + + local.resolve('prj_aaa').exists() + local.resolve('prj_aaa/file1.txt').text == 'Hello' + local.resolve('prj_aaa/file2.log').text == 'World' + + local.resolve('prj_bbb').exists() + local.resolve('prj_bbb/file1.txt').text == 'Ciao' + local.resolve('prj_bbb/file2.log').text == 'Mondo' } @@ -139,8 +159,9 @@ class UpdateModuleTest extends Specification { def target = baseFolder.resolve('target2') AssetManager.root = target.toFile() - when: + when:'legacy' def manager = new AssetManager( "file:${baseFolder}/pipe_2" ) + manager.setStrategyType(AssetManager.RepositoryStrategyType.LEGACY) manager.download() manager.updateModules() @@ -151,6 +172,21 @@ class UpdateModuleTest extends Specification { target.resolve('local/pipe_2/prj_aaa').list().size()==0 target.resolve('local/pipe_2/prj_bbb').list().size()==0 + + when: 'multi-revision' + manager = new AssetManager( "file:${baseFolder}/pipe_2" ) + manager.setStrategyType(AssetManager.RepositoryStrategyType.MULTI_REVISION) + manager.download() + manager.updateModules() + + then: + def local = manager.getLocalPath().toPath() + local.exists() + local.resolve('.git').exists() + local.resolve('main.nf').exists() + + local.resolve('prj_aaa').list().size()==0 + local.resolve('prj_bbb').list().size()==0 } def 'should clone selected submodules' () { @@ -174,6 +210,7 @@ class UpdateModuleTest extends Specification { when: def manager = new AssetManager( "file:${baseFolder}/pipe_3" ) + manager.setStrategyType(AssetManager.RepositoryStrategyType.LEGACY) manager.download() manager.updateModules() @@ -190,6 +227,26 @@ class UpdateModuleTest extends Specification { target.resolve('local/pipe_3/prj_ccc').exists() target.resolve('local/pipe_3/prj_ccc/file-x.txt').text == 'x' + when: 'multi-revision' + manager = new AssetManager( "file:${baseFolder}/pipe_3" ) + manager.setStrategyType(AssetManager.RepositoryStrategyType.MULTI_REVISION) + manager.download() + manager.updateModules() + + then: + def local = manager.getLocalPath().toPath() + local.exists() + local.resolve('.git').exists() + local.resolve('main.nf').exists() + + local.resolve('prj_aaa').list().size()==0 + + local.resolve('prj_bbb').exists() + local.resolve('prj_bbb/file1.txt').text == 'Ciao' + + local.resolve('prj_ccc').exists() + local.resolve('prj_ccc/file-x.txt').text == 'x' + } diff --git a/plugins/nf-k8s/src/main/nextflow/k8s/K8sDriverLauncher.groovy b/plugins/nf-k8s/src/main/nextflow/k8s/K8sDriverLauncher.groovy index c56a684cc6..d9a9f53048 100644 --- a/plugins/nf-k8s/src/main/nextflow/k8s/K8sDriverLauncher.groovy +++ b/plugins/nf-k8s/src/main/nextflow/k8s/K8sDriverLauncher.groovy @@ -271,7 +271,7 @@ class K8sDriverLauncher { if( !interactive && !pipelineName.startsWith('/') && !cmd.remoteProfile && !cmd.runRemoteConfig ) { // -- check and parse project remote config - final pipelineConfig = new AssetManager(pipelineName, cmd) .getConfigFile() + final pipelineConfig = new AssetManager(pipelineName, cmd).setRevision(cmd.revision).getConfigFile() builder.setUserConfigFiles(pipelineConfig) }