-
Notifications
You must be signed in to change notification settings - Fork 971
proposal: add artifact graph core query system #400
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
14 commits
Select commit
Hold shift + click to select a range
a94c819
proposal: add artifact graph core query system
TabishB e4f32fa
docs: specify Zod for schema validation in artifact graph proposal
TabishB 5f8cb32
docs: add 2-level schema resolution and built-in schemas
TabishB a835b4f
experiment: add vertical slice version of artifact graph change
TabishB 01cc5ce
feat(core): add getGlobalDataDir for XDG-compliant data directory
TabishB 8a222ab
feat(artifact-graph): add core dependency graph module
TabishB 87e7987
test(artifact-graph): add comprehensive test suite
TabishB b9d2261
docs(openspec): archive add-artifact-graph-core change
TabishB 392cf7a
chore: remove experimental artifact-graph-core-v2 folder
TabishB 1824297
feat(artifact-graph): validate global schema overrides
TabishB c73a691
test(artifact-graph): add workflow integration tests
TabishB f7de12b
refactor(artifact-graph): adopt zod v4 error message format
TabishB c51ba91
fix(test): prevent hanging vitest threads after test runs
TabishB 9a611ef
Merge branch 'main' into openspec/add-artifact-graph-core
TabishB File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Some comments aren't visible on the classic Files Changed page.
There are no files selected for viewing
197 changes: 197 additions & 0 deletions
197
openspec/changes/archive/2025-12-24-add-artifact-graph-core/design.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,197 @@ | ||
| ## Context | ||
|
|
||
| This implements "Slice 1: What's Ready?" from the artifact POC analysis. The core insight is using the filesystem as a database - artifact completion is detected by file existence, making the system stateless and version-control friendly. | ||
|
|
||
| This module will coexist with the current OpenSpec system as a parallel capability, potentially enabling future migration or integration. | ||
|
|
||
| ## Goals / Non-Goals | ||
|
|
||
| **Goals:** | ||
| - Pure dependency graph logic with no side effects | ||
| - Stateless state detection (rescan filesystem each query) | ||
| - Support glob patterns for multi-file artifacts (e.g., `specs/*.md`) | ||
| - Load artifact definitions from YAML schemas | ||
| - Calculate topological build order | ||
| - Determine "ready" artifacts based on dependency completion | ||
|
|
||
| **Non-Goals:** | ||
| - CLI commands (Slice 4) | ||
| - Multi-change management (Slice 2) | ||
| - Template resolution and enrichment (Slice 3) | ||
| - Agent integration or Claude commands | ||
| - Replacing existing OpenSpec functionality | ||
|
|
||
| ## Decisions | ||
|
|
||
| ### Decision: Filesystem as Database | ||
| Use file existence for state detection rather than a separate state file. | ||
|
|
||
| **Rationale:** | ||
| - Stateless - no state corruption possible | ||
| - Git-friendly - state derived from committed files | ||
| - Simple - no sync issues between state file and actual files | ||
|
|
||
| **Alternatives considered:** | ||
| - JSON/SQLite state file: More complex, sync issues, not git-friendly | ||
| - Git metadata: Too coupled to git, complex implementation | ||
|
|
||
| ### Decision: Kahn's Algorithm for Topological Sort | ||
| Use Kahn's algorithm for computing build order. | ||
|
|
||
| **Rationale:** | ||
| - Well-understood, O(V+E) complexity | ||
| - Naturally detects cycles during execution | ||
| - Produces a stable, deterministic order | ||
|
|
||
| ### Decision: Glob Pattern Support | ||
| Support glob patterns like `specs/*.md` in artifact `generates` field. | ||
|
|
||
| **Rationale:** | ||
| - Allows multiple files to satisfy a single artifact requirement | ||
| - Common pattern for spec directories with multiple files | ||
| - Uses standard glob syntax | ||
|
|
||
| ### Decision: Immutable Completed Set | ||
| Represent completion state as an immutable Set of completed artifact IDs. | ||
|
|
||
| **Rationale:** | ||
| - Functional style, easier to reason about | ||
| - State derived fresh each query, no mutation needed | ||
| - Clear separation between graph structure and runtime state | ||
| - Filesystem can only detect binary existence (complete vs not complete) | ||
|
|
||
| **Note:** `inProgress` and `failed` states are deferred to future slices. They would require external state tracking (e.g., a status file) since file existence alone cannot distinguish these states. | ||
|
|
||
| ### Decision: Zod for Schema Validation | ||
| Use Zod for validating YAML schema structure and deriving TypeScript types. | ||
|
|
||
| **Rationale:** | ||
| - Already a project dependency (v4.0.17) used in `src/core/schemas/` | ||
| - Type inference via `z.infer<>` - single source of truth for types | ||
| - Runtime validation with detailed error messages | ||
| - Consistent with existing project patterns (`base.schema.ts`, `config-schema.ts`) | ||
|
|
||
| **Alternatives considered:** | ||
| - Manual validation: More code, error-prone, no type inference | ||
| - JSON Schema: Would require additional dependency, less TypeScript integration | ||
| - io-ts: Not already in project, steeper learning curve | ||
|
|
||
| ### Decision: Two-Level Schema Resolution | ||
| Schemas resolve from global user data directory, falling back to package built-ins. | ||
|
|
||
| **Resolution order:** | ||
| 1. `${XDG_DATA_HOME:-~/.local/share}/openspec/schemas/<name>.yaml` - Global user override | ||
| 2. `<package>/schemas/<name>.yaml` - Built-in defaults | ||
|
|
||
| **Rationale:** | ||
| - Follows XDG Base Directory Specification (schemas are data, not config) | ||
| - Mirrors existing `getGlobalConfigDir()` pattern in `src/core/global-paths.ts` | ||
| - Built-ins baked into package, never auto-copied | ||
| - Users customize by creating files in global data dir | ||
| - Simple - no project-level overrides (can add later if needed) | ||
|
|
||
| **XDG compliance:** | ||
| - Uses `XDG_DATA_HOME` env var when set (all platforms) | ||
| - Unix/macOS fallback: `~/.local/share/openspec/` | ||
| - Windows fallback: `%LOCALAPPDATA%/openspec/` | ||
|
|
||
| **Alternatives considered:** | ||
| - Project-level overrides: Added complexity, not needed initially | ||
| - Auto-copy to user space: Creates drift, harder to update defaults | ||
| - Config directory (`XDG_CONFIG_HOME`): Schemas are workflow definitions (data), not user preferences (config) | ||
|
|
||
| ### Decision: Template Field Parsed But Not Resolved | ||
| The `template` field is required in schema YAML for completeness, but template resolution is deferred to Slice 3. | ||
|
|
||
| **Rationale:** | ||
| - Slice 1 focuses on "What's Ready?" - dependency and completion queries only | ||
| - Template paths are validated syntactically (non-empty string) but not resolved | ||
| - Keeps Slice 1 focused and independently testable | ||
|
|
||
| ### Decision: Cycle Error Format | ||
| Cycle errors list all artifact IDs in the cycle for easy debugging. | ||
|
|
||
| **Format:** `"Cyclic dependency detected: A → B → C → A"` | ||
|
|
||
| **Rationale:** | ||
| - Shows the full cycle path, not just that a cycle exists | ||
| - Actionable - developer can see exactly which artifacts to fix | ||
| - Consistent with Kahn's algorithm which naturally identifies cycle participants | ||
|
|
||
| ## Data Structures | ||
|
|
||
| **Zod Schemas (source of truth):** | ||
|
|
||
| ```typescript | ||
| import { z } from 'zod'; | ||
|
|
||
| // Artifact definition schema | ||
| export const ArtifactSchema = z.object({ | ||
| id: z.string().min(1, 'Artifact ID is required'), | ||
| generates: z.string().min(1), // e.g., "proposal.md" or "specs/*.md" | ||
| description: z.string(), | ||
| template: z.string(), // path to template file | ||
| requires: z.array(z.string()).default([]), | ||
| }); | ||
|
|
||
| // Full schema YAML structure | ||
| export const SchemaYamlSchema = z.object({ | ||
| name: z.string().min(1, 'Schema name is required'), | ||
| version: z.number().int().positive(), | ||
| description: z.string().optional(), | ||
| artifacts: z.array(ArtifactSchema).min(1, 'At least one artifact required'), | ||
| }); | ||
|
|
||
| // Derived TypeScript types | ||
| export type Artifact = z.infer<typeof ArtifactSchema>; | ||
| export type SchemaYaml = z.infer<typeof SchemaYamlSchema>; | ||
| ``` | ||
|
|
||
| **Runtime State (not Zod - internal only):** | ||
|
|
||
| ```typescript | ||
| // Slice 1: Simple completion tracking via filesystem | ||
| type CompletedSet = Set<string>; | ||
|
|
||
| // Return type for blocked query | ||
| interface BlockedArtifacts { | ||
| [artifactId: string]: string[]; // artifact → list of unmet dependencies | ||
| } | ||
|
|
||
| interface ArtifactGraphResult { | ||
| completed: string[]; | ||
| ready: string[]; | ||
| blocked: BlockedArtifacts; | ||
| buildOrder: string[]; | ||
| } | ||
| ``` | ||
|
|
||
| ## File Structure | ||
|
|
||
| ``` | ||
| src/core/artifact-graph/ | ||
| ├── index.ts # Public exports | ||
| ├── types.ts # Zod schemas and type definitions | ||
| ├── graph.ts # ArtifactGraph class | ||
| ├── state.ts # State detection logic | ||
| ├── resolver.ts # Schema resolution (global → built-in) | ||
| └── schemas/ # Built-in schema definitions (package level) | ||
| ├── spec-driven.yaml # Default: proposal → specs → design → tasks | ||
| └── tdd.yaml # Alternative: tests → implementation → docs | ||
| ``` | ||
|
|
||
| **Schema Resolution Paths:** | ||
| - Global user override: `${XDG_DATA_HOME:-~/.local/share}/openspec/schemas/<name>.yaml` | ||
| - Package built-in: `src/core/artifact-graph/schemas/<name>.yaml` (bundled with package) | ||
|
|
||
| ## Risks / Trade-offs | ||
|
|
||
| | Risk | Mitigation | | ||
| |------|------------| | ||
| | Glob pattern edge cases | Use well-tested glob library (fast-glob or similar) | | ||
| | Cycle detection | Kahn's algorithm naturally fails on cycles; provide clear error | | ||
| | Schema evolution | Version field in schema, validate on load | | ||
|
|
||
| ## Open Questions | ||
|
|
||
| None - all questions resolved in Decisions section. |
18 changes: 18 additions & 0 deletions
18
openspec/changes/archive/2025-12-24-add-artifact-graph-core/proposal.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,18 @@ | ||
| ## Why | ||
|
|
||
| The current OpenSpec system relies on conventions and AI inference for artifact ordering. A formal artifact graph with dependency awareness would enable deterministic "what's ready?" queries, making the system more predictable and enabling future features like automated pipeline execution. | ||
|
|
||
| ## What Changes | ||
|
|
||
| - Add `ArtifactGraph` class to model artifacts as a DAG with dependency relationships | ||
| - Add `ArtifactState` type to track completion status (completed, in_progress, failed) | ||
| - Add filesystem-based state detection using file existence and glob patterns | ||
| - Add schema YAML parser to load artifact definitions | ||
| - Implement topological sort (Kahn's algorithm) for build order calculation | ||
| - Add `getNextArtifacts()` to find artifacts ready for creation | ||
|
|
||
| ## Impact | ||
|
|
||
| - Affected specs: New `artifact-graph` capability | ||
| - Affected code: `src/core/artifact-graph/` (new directory) | ||
| - No changes to existing functionality - this is a parallel module | ||
103 changes: 103 additions & 0 deletions
103
...changes/archive/2025-12-24-add-artifact-graph-core/specs/artifact-graph/spec.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,103 @@ | ||
| ## ADDED Requirements | ||
|
|
||
| ### Requirement: Schema Loading | ||
| The system SHALL load artifact graph definitions from YAML schema files. | ||
|
|
||
| #### Scenario: Valid schema loaded | ||
| - **WHEN** a valid schema YAML file is provided | ||
| - **THEN** the system returns an ArtifactGraph with all artifacts and dependencies | ||
|
|
||
| #### Scenario: Invalid schema rejected | ||
| - **WHEN** a schema YAML file is missing required fields | ||
| - **THEN** the system throws an error with a descriptive message | ||
|
|
||
| #### Scenario: Cyclic dependencies detected | ||
| - **WHEN** a schema contains cyclic artifact dependencies | ||
| - **THEN** the system throws an error listing the artifact IDs in the cycle | ||
|
|
||
| #### Scenario: Invalid dependency reference | ||
| - **WHEN** an artifact's `requires` array references a non-existent artifact ID | ||
| - **THEN** the system throws an error identifying the invalid reference | ||
|
|
||
| #### Scenario: Duplicate artifact IDs rejected | ||
| - **WHEN** a schema contains multiple artifacts with the same ID | ||
| - **THEN** the system throws an error identifying the duplicate | ||
|
|
||
| ### Requirement: Build Order Calculation | ||
| The system SHALL compute a valid topological build order for artifacts. | ||
|
|
||
| #### Scenario: Linear dependency chain | ||
| - **WHEN** artifacts form a linear chain (A → B → C) | ||
| - **THEN** getBuildOrder() returns [A, B, C] | ||
|
|
||
| #### Scenario: Diamond dependency | ||
| - **WHEN** artifacts form a diamond (A → B, A → C, B → D, C → D) | ||
| - **THEN** getBuildOrder() returns A before B and C, and D last | ||
|
|
||
| #### Scenario: Independent artifacts | ||
| - **WHEN** artifacts have no dependencies | ||
| - **THEN** getBuildOrder() returns them in a stable order | ||
|
|
||
| ### Requirement: State Detection | ||
| The system SHALL detect artifact completion state by scanning the filesystem. | ||
|
|
||
| #### Scenario: Simple file exists | ||
| - **WHEN** an artifact generates "proposal.md" and the file exists | ||
| - **THEN** the artifact is marked as completed | ||
|
|
||
| #### Scenario: Simple file missing | ||
| - **WHEN** an artifact generates "proposal.md" and the file does not exist | ||
| - **THEN** the artifact is not marked as completed | ||
|
|
||
| #### Scenario: Glob pattern with files | ||
| - **WHEN** an artifact generates "specs/*.md" and the specs/ directory contains .md files | ||
| - **THEN** the artifact is marked as completed | ||
|
|
||
| #### Scenario: Glob pattern empty | ||
| - **WHEN** an artifact generates "specs/*.md" and the specs/ directory is empty or missing | ||
| - **THEN** the artifact is not marked as completed | ||
|
|
||
| #### Scenario: Missing change directory | ||
| - **WHEN** the change directory does not exist | ||
| - **THEN** all artifacts are marked as not completed (empty state) | ||
|
|
||
| ### Requirement: Ready Artifact Query | ||
| The system SHALL identify which artifacts are ready to be created based on dependency completion. | ||
|
|
||
| #### Scenario: Root artifacts ready initially | ||
| - **WHEN** no artifacts are completed | ||
| - **THEN** getNextArtifacts() returns artifacts with no dependencies | ||
|
|
||
| #### Scenario: Dependent artifact becomes ready | ||
| - **WHEN** an artifact's dependencies are all completed | ||
| - **THEN** getNextArtifacts() includes that artifact | ||
|
|
||
| #### Scenario: Blocked artifacts excluded | ||
| - **WHEN** an artifact has uncompleted dependencies | ||
| - **THEN** getNextArtifacts() does not include that artifact | ||
|
|
||
| ### Requirement: Completion Check | ||
| The system SHALL determine when all artifacts in a graph are complete. | ||
|
|
||
| #### Scenario: All complete | ||
| - **WHEN** all artifacts in the graph are in the completed set | ||
| - **THEN** isComplete() returns true | ||
|
|
||
| #### Scenario: Partially complete | ||
| - **WHEN** some artifacts in the graph are not completed | ||
| - **THEN** isComplete() returns false | ||
|
|
||
| ### Requirement: Blocked Query | ||
| The system SHALL identify which artifacts are blocked and return all their unmet dependencies. | ||
|
|
||
| #### Scenario: Artifact blocked by single dependency | ||
| - **WHEN** artifact B requires artifact A and A is not complete | ||
| - **THEN** getBlocked() returns `{ B: ['A'] }` | ||
|
|
||
| #### Scenario: Artifact blocked by multiple dependencies | ||
| - **WHEN** artifact C requires A and B, and only A is complete | ||
| - **THEN** getBlocked() returns `{ C: ['B'] }` | ||
|
|
||
| #### Scenario: Artifact blocked by all dependencies | ||
| - **WHEN** artifact C requires A and B, and neither is complete | ||
| - **THEN** getBlocked() returns `{ C: ['A', 'B'] }` |
61 changes: 61 additions & 0 deletions
61
openspec/changes/archive/2025-12-24-add-artifact-graph-core/tasks.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,61 @@ | ||
| ## 1. Type Definitions | ||
| - [x] 1.1 Create `src/core/artifact-graph/types.ts` with Zod schemas (`ArtifactSchema`, `SchemaYamlSchema`) and inferred types via `z.infer<>` | ||
| - [x] 1.2 Define `CompletedSet` (Set<string>), `BlockedArtifacts`, and `ArtifactGraphResult` types for runtime state | ||
|
|
||
| ## 2. Schema Parser | ||
| - [x] 2.1 Create `src/core/artifact-graph/schema.ts` with YAML loading and Zod validation via `.safeParse()` | ||
| - [x] 2.2 Implement dependency reference validation (ensure `requires` references valid artifact IDs) | ||
| - [x] 2.3 Implement duplicate artifact ID detection | ||
| - [x] 2.4 Add cycle detection during schema load (error format: "Cyclic dependency detected: A → B → C → A") | ||
|
|
||
| ## 3. Artifact Graph Core | ||
| - [x] 3.1 Create `src/core/artifact-graph/graph.ts` with ArtifactGraph class | ||
| - [x] 3.2 Implement `fromYaml(path)` - load graph from schema file | ||
| - [x] 3.3 Implement `getBuildOrder()` - topological sort via Kahn's algorithm | ||
| - [x] 3.4 Implement `getArtifact(id)` - retrieve single artifact definition | ||
| - [x] 3.5 Implement `getAllArtifacts()` - list all artifacts | ||
|
|
||
| ## 4. State Detection | ||
| - [x] 4.1 Create `src/core/artifact-graph/state.ts` with state detection logic | ||
| - [x] 4.2 Implement file existence checking for simple paths | ||
| - [x] 4.3 Implement glob pattern matching for multi-file artifacts | ||
| - [x] 4.4 Implement `detectCompleted(graph, changeDir)` - scan filesystem and return CompletedSet | ||
| - [x] 4.5 Handle missing changeDir gracefully (return empty CompletedSet) | ||
|
|
||
| ## 5. Ready Calculation | ||
| - [x] 5.1 Implement `getNextArtifacts(graph, completed)` - find artifacts with all deps completed | ||
| - [x] 5.2 Implement `isComplete(graph, completed)` - check if all artifacts done | ||
| - [x] 5.3 Implement `getBlocked(graph, completed)` - return BlockedArtifacts map (artifact → unmet deps) | ||
|
|
||
| ## 6. Schema Resolution | ||
| - [x] 6.1 Create `src/core/artifact-graph/resolver.ts` with schema resolution logic | ||
| - [x] 6.2 Add `getGlobalDataDir()` to `src/core/global-config.ts` (XDG_DATA_HOME with platform fallbacks) | ||
| - [x] 6.3 Implement `resolveSchema(name)` - global (`${XDG_DATA_HOME}/openspec/schemas/`) → built-in fallback | ||
|
|
||
| ## 7. Built-in Schemas | ||
| - [x] 7.1 Create `src/core/artifact-graph/schemas/spec-driven.yaml` (default: proposal → specs → design → tasks) | ||
| - [x] 7.2 Create `src/core/artifact-graph/schemas/tdd.yaml` (alternative: tests → implementation → docs) | ||
|
|
||
| ## 8. Integration | ||
| - [x] 8.1 Create `src/core/artifact-graph/index.ts` with public exports | ||
|
|
||
| ## 9. Testing | ||
| - [x] 9.1 Test: Parse valid schema YAML returns correct artifact graph | ||
| - [x] 9.2 Test: Parse invalid schema (missing fields) throws descriptive error | ||
| - [x] 9.3 Test: Duplicate artifact IDs throws error | ||
| - [x] 9.4 Test: Invalid `requires` reference throws error identifying the invalid ID | ||
| - [x] 9.5 Test: Cycle in schema throws error listing cycle path (e.g., "A → B → C → A") | ||
| - [x] 9.6 Test: Compute build order returns correct topological ordering (linear chain) | ||
| - [x] 9.7 Test: Compute build order handles diamond dependencies correctly | ||
| - [x] 9.8 Test: Independent artifacts return in stable order | ||
| - [x] 9.9 Test: Empty/missing changeDir returns empty CompletedSet | ||
| - [x] 9.10 Test: File existence marks artifact as completed | ||
| - [x] 9.11 Test: Glob pattern specs/*.md detected as complete when files exist | ||
| - [x] 9.12 Test: Glob pattern with empty directory not marked complete | ||
| - [x] 9.13 Test: getNextArtifacts returns only root artifacts when nothing completed | ||
| - [x] 9.14 Test: getNextArtifacts includes artifact when all deps completed | ||
| - [x] 9.15 Test: getBlocked returns artifact with all unmet dependencies listed | ||
| - [x] 9.16 Test: isComplete() returns true when all artifacts completed | ||
| - [x] 9.17 Test: isComplete() returns false when some artifacts incomplete | ||
| - [x] 9.18 Test: Schema resolution finds global override before built-in | ||
| - [x] 9.19 Test: Schema resolution falls back to built-in when no global |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.