feat: add map and array steps for parallel processing #207

jumski · 2025-09-10T11:38:33Z

Summary

Adds a new .array() method to the Flow class that provides compile-time type safety for array-returning step handlers while maintaining full compatibility with existing functionality.

Key Features

Compile-time type enforcement: TypeScript rejects non-array return types
Zero runtime overhead: Pure delegation to existing .step() method
Full feature parity: Complete support for dependencies, runtime options, and validation
Semantic clarity: Clear intent when creating array-producing steps

// Type safety - these will fail at compile time:
flow.array({ slug: 'invalid' }, () => 42);        // ❌ Error: not an array
flow.array({ slug: 'invalid2' }, () => "string"); // ❌ Error: not an array

Implementation Details

Pure delegation: return this.step(opts, handler); for maximum code reuse
Type constraints: Array<Json> | Promise<Array<Json>> return type enforcement
Utility type reuse: Leverages existing StepInput<this, Deps>, AwaitedReturn<THandler>, etc.
Identical behavior: Produces the same step definitions as .step() for array handlers

Testing

169 total tests passing (162 existing + 7 new integration tests)
15 type tests validating compile-time constraints
27 runtime tests ensuring behavioral equivalence with .step()
7 integration tests covering complex real-world scenarios

Breaking Changes

None. This is purely additive functionality that maintains full backward compatibility.

Review skipped

Auto reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

✨ Finishing touches

🧪 Generate unit tests

Create PR with unit tests
Post copyable unit tests in a comment
Commit unit tests in branch feature-map-and-array

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

jumski · 2025-09-10T11:38:55Z

How to use the Graphite Merge Queue

Add either label to this PR to merge it via the merge queue:

merge:queue - adds this PR to the back of the merge queue
hotfix:queue - for urgent hot fixes, skip the queue and merge this PR next

You must have a Graphite account in order to use the merge queue. Sign up using this link.

_{An organization admin has enabled the Graphite Merge Queue in this repository.} _{Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue.}

This stack of pull requests is managed by Graphite. Learn more about stacking.

nx-cloud · 2025-09-10T11:45:26Z

View your CI Pipeline Execution ↗ for commit dfed36c

Command	Status	Duration	Result
`nx run-many -t build --projects client,dsl --co...`	✅ Succeeded	3s	View ↗
`nx affected -t build --configuration=production...`	✅ Succeeded	3s	View ↗
`nx affected -t lint typecheck test --parallel -...`	✅ Succeeded	6m 5s	View ↗

☁️ Nx Cloud last updated this comment at 2025-10-06 15:33:33 UTC

…ow steps Implement a new .array() method in the Flow class to enable type-safe, semantic array steps that leverage existing validation and dependency mechanisms. The change includes type constraints to enforce array return types, comprehensive runtime and type tests, and detailed documentation. This enhancement maintains full feature parity with .step() and integrates seamlessly with existing validation utilities, laying the foundation for future phases involving queue routing and advanced type inference. The implementation follows the principle of minimal, non-breaking extension, ensuring zero risk to current functionality and providing immediate compile-time safety benefits.

github-actions · 2025-10-06T15:33:28Z

🔍 Preview Deployment: Website

✅ Deployment successful!

🔗 Preview URL: https://pr-207.pgflow.pages.dev

📝 Details:

Branch: feature-map-and-array
Commit: 3a9f5885420bb24c2f0dc28dd7b5bf11d0ba5082
View Logs

_Last updated: _

github-actions · 2025-10-06T15:35:14Z

🔍 Preview Deployment: Playground

✅ Deployment successful!

🔗 Preview URL: https://pr-207--pgflow-demo.netlify.app

📝 Details:

Branch: feature-map-and-array
Commit: 3a9f5885420bb24c2f0dc28dd7b5bf11d0ba5082
View Logs

_Last updated: _

graphite-app · 2025-10-07T08:28:17Z

Merge activity

Oct 7, 8:28 AM UTC: jumski added this pull request to the Graphite merge queue.
Oct 7, 8:28 AM UTC: CI is running for this pull request on a draft pull request (#232) due to your merge queue CI optimization settings.
Oct 7, 8:29 AM UTC: Merged by the Graphite merge queue via draft PR: #232.

## Summary Adds a new `.array()` method to the Flow class that provides compile-time type safety for array-returning step handlers while maintaining full compatibility with existing functionality. ## Key Features - **Compile-time type enforcement**: TypeScript rejects non-array return types - **Zero runtime overhead**: Pure delegation to existing `.step()` method - **Full feature parity**: Complete support for dependencies, runtime options, and validation - **Semantic clarity**: Clear intent when creating array-producing steps ```ts // Type safety - these will fail at compile time: flow.array({ slug: 'invalid' }, () => 42); // ❌ Error: not an array flow.array({ slug: 'invalid2' }, () => "string"); // ❌ Error: not an array ``` ## Implementation Details - **Pure delegation**: `return this.step(opts, handler);` for maximum code reuse - **Type constraints**: `Array<Json> | Promise<Array<Json>>` return type enforcement - **Utility type reuse**: Leverages existing `StepInput<this, Deps>`, `AwaitedReturn<THandler>`, etc. - **Identical behavior**: Produces the same step definitions as `.step()` for array handlers ## Testing - **169 total tests passing** (162 existing + 7 new integration tests) - **15 type tests** validating compile-time constraints - **27 runtime tests** ensuring behavioral equivalence with `.step()` - **7 integration tests** covering complex real-world scenarios ## Breaking Changes None. This is purely additive functionality that maintains full backward compatibility. ## Related This implements Phase 1 of the array processing roadmap from PLAN.md, laying the foundation for future phases including queue routing and `.map()` method integration.

jumski marked this pull request as ready for review September 10, 2025 11:39

jumski changed the title ~~feat(dsl): add .array() method for type-safe array creation in workflow steps~~ feat: add map and array steps for parallel processing Sep 10, 2025

jumski temporarily deployed to preview September 10, 2025 11:45 — with GitHub Actions Inactive

jumski force-pushed the feature-map-and-array branch from 981dc83 to 08a197e Compare September 10, 2025 12:24

jumski temporarily deployed to preview September 10, 2025 12:31 — with GitHub Actions Inactive

This was referenced Sep 10, 2025

feat: add map step type in sql #208

Closed

feat: add root map step support #209

Closed

jumski force-pushed the feature-map-and-array branch from 08a197e to a0af828 Compare September 11, 2025 16:04

jumski temporarily deployed to preview September 11, 2025 16:11 — with GitHub Actions Inactive

jumski force-pushed the feature-map-and-array branch from a0af828 to c66f5b6 Compare September 12, 2025 07:59

This was referenced Sep 12, 2025

# 🚀 Task Spawning for Map Steps #213

Closed

feat: implement cascade completion of taskless steps #214

Closed

jumski force-pushed the feature-map-and-array branch from c66f5b6 to 436b03d Compare September 15, 2025 17:54

jumski temporarily deployed to preview September 15, 2025 18:01 — with GitHub Actions Inactive

jumski force-pushed the feature-map-and-array branch from 436b03d to ee33c4a Compare October 5, 2025 18:59

jumski mentioned this pull request Oct 5, 2025

docs: improve documentation and READMEs to explain map/array steps #228

Closed

jumski force-pushed the feature-map-and-array branch from ee33c4a to 2d741f6 Compare October 5, 2025 19:20

jumski temporarily deployed to preview October 6, 2025 04:04 — with GitHub Actions Inactive

jumski mentioned this pull request Oct 6, 2025

fix: remove temp migrations and prepare stack to be merged #230

Closed

jumski force-pushed the feature-map-and-array branch from 2d741f6 to dfed36c Compare October 6, 2025 15:25

jumski temporarily deployed to preview October 6, 2025 15:32 — with GitHub Actions Inactive

jumski mentioned this pull request Oct 6, 2025

fix: typecheck tests using strict tsc mode #231

Closed

graphite-app bot closed this Oct 7, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: add map and array steps for parallel processing #207

feat: add map and array steps for parallel processing #207

Uh oh!

jumski commented Sep 10, 2025 •

edited

Loading

Uh oh!

changeset-bot bot commented Sep 10, 2025 •

edited

Loading

Uh oh!

coderabbitai bot commented Sep 10, 2025 •

edited

Loading

Review skipped

Uh oh!

jumski commented Sep 10, 2025 •

edited

Loading

Uh oh!

nx-cloud bot commented Sep 10, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Oct 6, 2025

Uh oh!

github-actions bot commented Oct 6, 2025

Uh oh!

graphite-app bot commented Oct 7, 2025 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

feat: add map and array steps for parallel processing #207

feat: add map and array steps for parallel processing #207

Uh oh!

Conversation

jumski commented Sep 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Key Features

Implementation Details

Testing

Breaking Changes

Related

Uh oh!

changeset-bot bot commented Sep 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

coderabbitai bot commented Sep 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

jumski commented Sep 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

How to use the Graphite Merge Queue

Uh oh!

nx-cloud bot commented Sep 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Oct 6, 2025

🔍 Preview Deployment: Website

Uh oh!

github-actions bot commented Oct 6, 2025

🔍 Preview Deployment: Playground

Uh oh!

graphite-app bot commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge activity

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

jumski commented Sep 10, 2025 •

edited

Loading

changeset-bot bot commented Sep 10, 2025 •

edited

Loading

coderabbitai bot commented Sep 10, 2025 •

edited

Loading

jumski commented Sep 10, 2025 •

edited

Loading

nx-cloud bot commented Sep 10, 2025 •

edited

Loading

graphite-app bot commented Oct 7, 2025 •

edited

Loading