Implementation of multiple revisions without revisions map

adr/20251205-multi-revision-asset-management-strategy-pattern.md

@@ -0,0 +1,262 @@
+# Multi-Revision Asset Management with Strategy Pattern
+
+- Authors: Jorge Ejarque
+- Status: Pending
+- Deciders: Jorge Ejarque, Ben Sherman, Paolo Di Tommaso
+- Date: 2025-12-05
+- Tags: scm, asset-management, multi-revision
+
+## Summary
+
+Nextflow's asset management system has been refactored to support multiple revisions of the same pipeline concurrently through a bare repository approach with shared object storage, while maintaining backward compatibility with legacy direct-clone repositories using the Strategy design pattern.
+
+## Problem Statement
+
+The original asset management system (`AssetManager`) cloned each pipeline directly to `~/.nextflow/assets/<org>/<project>/.git`, creating several limitations:
+
+1. **No concurrent multi-revision support**: Only one revision of a pipeline could be checked out at a time, preventing concurrent execution of different versions
+2. **Update conflicts**: Pulling updates while a pipeline was running could cause conflicts or corruption
+3. **Testing limitations**: Users couldn't easily test different versions of a pipeline side-by-side
+
+The goal was to enable running multiple revisions of the same pipeline concurrently (e.g., production on v1.0, testing on v2.0-dev) while maintaining efficient disk usage through object sharing.
+
+## Goals or Decision Drivers
+
+- **Concurrent multi-revision execution**: Must support running different revisions of the same pipeline simultaneously
+- **Efficient disk usage**: Share Git objects between revisions to minimize storage overhead
+- **Backward compatibility**: Must not break existing pipelines using the legacy direct-clone approach
+- **API stability**: Maintain the existing `AssetManager` API for external consumers (K8s plugin, CLI commands, etc.)
+- **Minimal migration impact**: Existing repositories should continue to work without user intervention
+- **JGit compatibility**: Solution must work within JGit's capabilities to avoid relying on Git client installations
+- **Atomic updates**: Downloading new revisions should not interfere with running pipelines
+
+## Non-goals
+
+- **Migration of existing legacy repositories**: Legacy repos continue to work as-is; no forced migration
+- **Native Git worktree support**: Due to JGit limitations, not using Git's worktree feature
+- **Revision garbage collection**: No automatic cleanup of old revisions (users can manually drop)
+- **Multi-hub support**: Still tied to a single repository provider per pipeline
+
+## Considered Options
+
+### Option 1: Bare Repository with Git Worktrees
+
+Use Git's worktree feature to create multiple working directories from a single bare repository.
+
+**Implementation**:
+- One bare repository at `~/.nextflow/assets/<org>/<project>/.git`
+- Multiple worktrees at `~/.nextflow/assets/<org>/<project>/<revision>/`
+
+- Good, because it's the native Git solution for multiple checkouts
+- Good, because worktrees are space-efficient
+- Good, because Git handles all the complexity
+- **Bad, because JGit doesn't support worktrees** (deal-breaker)
+- Bad, because requires native Git installation
+
+**Decision**: Rejected due to JGit incompatibility 
+
+### Option 2: Bare Repository + Clones per Commit + Revision Map File
+
+Use a bare repository for storage and create clones for each commit, tracking them in a separate file.
+
+**Implementation**:
+- Bare repository at `~/.nextflow/assets/<org>/<project>/.nextflow/bare_repo/`
+- Clones at `~/.nextflow/assets/<org>/<project>/.nextflow/commits/<commit-sha>/`
+- Revision map file at `~/.nextflow/assets/<org>/<project>/.nextflow/revisions.json` mapping revision names to commit SHAs
+
+- Good, because it works with JGit
+- Good, because bare repo reduces remote repository interactions to checkout commits
+- Good, because explicit revision tracking
+- Bad, because disk space as git objects replicated in clones 
+- Bad, because revision map file can become stale
+- Bad, because requires file I/O for every revision lookup
+- Bad, because potential race conditions on map file updates
+- Bad, because adds complexity of maintaining external state
+
+**Decision**: Initially implemented but later refined
+
+### Option 3: Bare Repository + Clones per Commit (Bare as Revision Map)
+
+Similar to Option 2 but create eliminate the separate revision map file by using the bare repository itself as the source of truth.
+
+**Implementation**:
+- Bare repository at `~/.nextflow/assets/.repos/<org>/<project>/.nextflow/bare/`
+- Shared clones at `~/.nextflow/assets/.repos/<org>/<project>/.nextflow/commits/<commit-sha>/`
+- Use bare repository refs to resolve revisions to commit SHAs dynamically
+- JGit alternates mechanism for object sharing
+
+- Good, because no external state file to maintain
+- Good, because bare repository is always in sync (fetched on updates)
+- Good, because simpler and more reliable
+- Good, because atomic updates (Git operations are atomic)
+- Good, because works entirely within JGit
+- Bad, because requires resolution on every access (minimal overhead)
+
+**Decision**: Selected as the multi-revision implementation
+
+### Option 4: Strategy Pattern for Backward Compatibility
+
+Instead of migrating existing repositories, use the Strategy pattern to support both legacy and multi-revision approaches.
+
+**Implementation**:
+- `AssetManager` as facade with unchanged public API
+- `RepositoryStrategy` interface defining repository operations
+- `LegacyRepositoryStrategy` for existing direct-clone behavior
+- `MultiRevisionRepositoryStrategy` for new bare-repo approach
+- Strategy selection based on environment variable or repository state detection
+
+- Good, because zero migration needed
+- Good, because maintains API compatibility
+- Good, because allows gradual adoption
+- Good, because isolates legacy code
+- Good, because makes future strategies easy to add
+- Neutral, because adds abstraction layer
+- Bad, because increases codebase size initially
+
+**Decision**: Selected for backward compatibility layer
+
+## Solution or decision outcome
+
+Implemented **Option 3 (Bare Repository + Shared Clones per Commit)** for multi-revision support, combined with **Option 4 (Strategy Pattern)** for backward compatibility. Multi-revision is the default for new repositories, while legacy mode is available via `NXF_SCM_LEGACY` environment variable.
+
+## Rationale & discussion
+
+### Multi-Revision Implementation (Option 3)
+
+The bare repository approach provides efficient multi-revision support:
+
+```
+~/.nextflow/assets/.repos/nextflow-io/hello/
+├── .nextflow/
+│   ├── bare_repo/              # Bare repository (shared objects)
+│   │   ├── objects/            # All Git objects stored here
+│   │   ├── refs/
+│   │   │   ├── heads/
+│   │   │   └── tags/
+│   │   └── config
+│   │
+│   └── commits/                # Commit-specific clones
+│       ├── abc123.../          # Clone for commit abc123
+│       │   └── .git/
+│       │       ├── objects/    # (uses alternates → bare_repo)
+│       │       └── info/
+│       │           └── alternates  # Points to bare_repo/objects
+│       │
+│       └── def456.../          # Clone for commit def456
+│           └── .git/
+└── .git/                       # Optional legacy repo (HYBRID state)
+```
+
+**Key mechanisms:**
+
+1. **Bare repository as source of truth**: The bare repo is fetched/updated from the remote, keeping refs current
+2. **Dynamic resolution**: Revisions (branch/tag names) are resolved to commit SHAs using the bare repo's refs
+3. **Object sharing**: Clones use Git alternates to reference the bare repo's objects, avoiding duplication
+4. **Atomic operations**: Each clone is independent; downloading a new revision doesn't affect existing ones
+5. **Lazy creation**: Clones are created on-demand when a specific revision is requested
+
+**Advantages over revision map file:**
+- No external state to maintain or keep in sync
+- Bare repo fetch automatically updates all refs
+- Resolution is simple: `bareRepo.resolve(revision)` returns commit SHA
+- No race conditions on file updates
+- Simpler code with fewer failure modes
+
+### Strategy Pattern Implementation (Option 4)
+
+The Strategy pattern provides clean separation and backward compatibility:
+
+```
+┌─────────────────────────┐
+│     AssetManager        │  ← Public API (unchanged)
+│     (Facade)            │
+└───────────┬─────────────┘
+            │
+            │ delegates to
+            ▼
+┌─────────────────────────┐
+│  RepositoryStrategy     │  ← Interface
+└───────────┬─────────────┘
+            △
+            │ implements
+    ┌───────┴────────┐
+    │                │
+┌───────────┐  ┌─────────────────┐
+│  Legacy   │  │ MultiRevision   │  ← Concrete strategies
+│ Strategy  │  │   Strategy      │
+└───────────┘  └─────────────────┘
+```
+
+**Strategy selection logic:**
+
+1. Check `NXF_SCM_LEGACY` environment variable → Use legacy if set
+2. Detect repository state:
+   - `UNINITIALIZED` (no repo) → Use multi-revision (default for new)
+   - `LEGACY_ONLY` (only `.git/`) → Use legacy (preserve existing)
+   - `BARE_ONLY` (only bare repo) → Use multi-revision
+   - `HYBRID` (both exist) → Prefer multi-revision
+
+**Backward compatibility guarantees:**
+
+- Existing repositories continue to work without changes
+- `AssetManager` API remains identical
+- CLI commands work with both strategies transparently
+- Tests pass with minimal modifications
+- No forced migration; users opt-in naturally when creating new repos
+
+### Hybrid State Handling
+
+The system gracefully handles hybrid states where both legacy and multi-revision repositories coexist:
+
+- **Detection**: `RepositoryStatus` enum represents all possible states
+- **Fallback logic**: Multi-revision strategy can fall back to legacy repo for operations if needed
+- **No conflicts**: Strategies are designed to coexist; operations target different directories
+- **Explicit control**: Users can force a specific strategy via `setStrategyType()`
+
+### Migration Path
+
+Users naturally migrate as they pull new revisions:
+
+1. **Existing users**: Continue with legacy repos (`LEGACY_ONLY` state detected)
+2. **New users**: Get multi-revision by default (`UNINITIALIZED` → multi-revision)
+3. **Opt-in migration**: Delete project directory to switch to multi-revision or pull with --migrate
+4. **Opt-out**: Set `NXF_SCM_LEGACY=true` to force legacy mode
+
+### Implementation Details
+
+**Key classes:**
+
+- `RepositoryStrategy`: Interface defining repository operations
+- `AbstractRepositoryStrategy`: Base class with shared helper methods
+- `LegacyRepositoryStrategy`: Direct clone implementation (original behavior)
+- `MultiRevisionRepositoryStrategy`: Bare repo + shared clones implementation
+
+**Critical methods:**
+
+- `download()`: Equivalent for both strategies (legacy pulls, multi-revision creates shared clone)
+- `getLocalPath()`: Returns appropriate working directory based on strategy
+- `getGit()`: Returns appropriate Git instance (legacy git, bare git, or commit git)
+
+### Performance Characteristics
+
+**Disk usage:**
+- Legacy: ~100% per repository (full clone with all git objects) + Worktree
+- Multi-revision: ~100% for bare + ~100K (.git with alternates) per revision + Worktree per revision
+
+**Operation speed:**
+- First download: Similar (both clone from remote)
+- Additional revisions: Multi-revision faster (only fetches new objects once, creates cheap clones)
+- Switching revisions: Multi-revision instant (different directories), legacy requires checkout
+
+### Known Limitations
+
+- No automatic migration of legacy repositories
+- Bare repository overhead even for users who only need one revision
+- JGit alternates slightly more complex than worktrees
+- Manual cleanup required for old revision clones
+
+## Links
+- [GitHub Issue #2870 - Multiple revisions of the same pipeline for concurrent execution](https://github.com/nextflow-io/nextflow/issues/2870)
+- [PR #6620 - Implementation of multiple revisions without revisions map](https://github.com/nextflow-io/nextflow/pull/6620)
+- Related PRs implementing the multi-revision approach (linked in #6620)
+

docs/cli.md

@@ -79,6 +79,13 @@ $ nextflow run nextflow-io/hello -r v1.1
 $ nextflow run nextflow-io/hello -r dev-branch
 $ nextflow run nextflow-io/hello -r a3f5c8e
 ```
+:::{versionadded} 25.12.0-edge
+:::
+Nextflow downloads and stores each explicitly requested Git branch, tag, or commit ID in a separate directory path, enabling you to run multiple revisions of the same pipeline simultaneously. Downloaded revisions are stored in a subdirectory of the local project: `$NXF_ASSETS/.repos/<org>/<repo>/.nextflow/commits/<commitId>`.
+
+:::{tip}
+Use tags or commit IDs instead of branches for reproducible pipeline runs. Branch references change as development progresses over time.
+:::
 
 (cli-params)=
 
@@ -176,7 +183,7 @@ main script : main.nf
 revisions   :
 * master (default)
   mybranch
-  v1.1 [t]
+* v1.1 [t]
   v1.2 [t]
 ```
 
@@ -186,7 +193,7 @@ This shows:
 - Where it's cached locally
 - Which script runs by default
 - Available revisions (branches and tags marked with `[t]`)
-- Which revision is currently checked out (marked with `*`)
+- Which revisions are currently pulled (marked with `*`)
 
 ### Pulling or updating projects
 

docs/developer/diagrams/nextflow.scm.mmd

@@ -8,19 +8,68 @@ classDiagram
 
     class AssetManager {
         project : String
-        localPath : File
         mainScript : String
-        repositoryProvider : RepositoryProvider
+        provider : RepositoryProvider
+        strategy : RepositoryStrategy
         hub : String
         providerConfigs : List~ProviderConfig~
     }
-    AssetManager --* RepositoryProvider
+    class RepositoryStatus {
+        <<enumeration>>
+        UNINITIALIZED
+        LEGACY_ONLY
+        BARE_ONLY
+        HYBRID
+    }
+    class RepositoryStrategyType {
+        <<enumeration>>
+        LEGACY
+        MULTI_REVISION
+    }
+
+    AssetManager --> RepositoryStatus
+    AssetManager --> RepositoryStrategyType
+    AssetManager "1" --o "1" RepositoryStrategy
+    AssetManager "1" --o "1" RepositoryProvider
     AssetManager "1" --* "*" ProviderConfig
 
+    class RepositoryStrategy {
+        <<interface>>
+    }
+    class AbstractRepositoryStrategy {
+        <<abstract>>
+        project : String
+        provider : RepositoryProvider
+        root : File
+    }
+    class LegacyRepositoryStrategy {
+        localPath : File
+    }
+    class MultiRevisionRepositoryStrategy {
+        revision : String
+        bareRepo : File
+        commitPath : File
+        revisionSubdir : File
+    }
+
+    RepositoryStrategy <|-- AbstractRepositoryStrategy
+    AbstractRepositoryStrategy <|-- LegacyRepositoryStrategy
+    AbstractRepositoryStrategy <|-- MultiRevisionRepositoryStrategy
+
+    class RepositoryProvider {
+        <<abstract>>
+    }
+
+    RepositoryStrategy --> RepositoryProvider
+
     RepositoryProvider <|-- AzureRepositoryProvider
     RepositoryProvider <|-- BitbucketRepositoryProvider
     RepositoryProvider <|-- BitbucketServerRepositoryProvider
     RepositoryProvider <|-- GiteaRepositoryProvider
     RepositoryProvider <|-- GithubRepositoryProvider
     RepositoryProvider <|-- GitlabRepositoryProvider
     RepositoryProvider <|-- LocalRepositoryProvider
+
+    class FileMutex
+
+    MultiRevisionRepositoryStrategy --> FileMutex