feat: Add `.map()` Method to Flow DSL for Map Step Support #218

jumski · 2025-09-18T06:26:23Z

Overview

This PR introduces the .map() method to the Flow DSL, enabling users to define map-type steps that process arrays element-by-element. This completes the DSL support for the map step infrastructure that was previously added to the SQL Core layer.

Problem

Previously, while the SQL Core supported map steps (with step_type => 'map'), there was no way to define them via the TypeScript DSL. Users had to manually write SQL or work around this limitation.

Solution

The new .map() method provides a type-safe, intuitive way to define map steps with these key characteristics:

API Design

// Root map - processes flow input array
new Flow<string[]>({ slug: 'text_processing' })
  .map({ slug: 'normalize' }, (text) => text.trim().toLowerCase())

// Dependent map - processes another step's array output
new Flow<{}>({ slug: 'workflow' })
  .array({ slug: 'fetch_items' }, () => fetchData())
  .map({ slug: 'enrich', array: 'fetch_items' }, (item) => ({
    ...item,
    enriched: true
  }))

Key Features

Intuitive dependency syntax
- Omit array property for root maps (processes flow input)
- Use array: 'stepSlug' for dependent maps (single dependency only)
Different handler signature
- Map handler: (item, context) => Json
- Regular step handler: (input, context) => Json
- Map handlers receive individual array elements, not the full input object
Type safety
- Enforces Json-compatible types for inputs and outputs
- Full TypeScript inference for item types based on dependencies
- Compile-time validation that dependencies return arrays
SQL generation
- Correctly adds step_type => 'map' parameter to pgflow.add_step()
- Maintains proper parameter ordering: dependencies, options, then step_type

Implementation Details

Changes Made

Extended StepDefinition interface (dsl.ts)
- Added optional stepType?: 'single' | 'map' property
- Defaults to 'single' for backward compatibility
Implemented .map() method (dsl.ts)
- Complex type signatures for proper inference
- Runtime validation for dependencies
- Stores stepType: 'map' in step definition
Updated compile function (compile-flow.ts)
- Checks for stepType property
- Adds step_type => 'map' parameter when needed
Comprehensive test coverage
- Type tests: 20 tests covering constraints and inference
- Runtime tests: 12 tests for SQL generation
- Integration tests: 12 tests for complete workflows
- Example flows demonstrating various use cases

Technical Decisions

Optional array key: Mirrors the dependsOn pattern (no key = root step)
Single dependency constraint: Enforced at both type and runtime levels
Handler signature difference: Clear distinction between map and regular steps
Type parameter ordering: Worked around TypeScript constraints for optional parameters

Testing

The implementation follows TDD principles with:

✅ Type safety tests (inference, constraints, edge cases)
✅ SQL generation tests (parameter ordering, step_type inclusion)
✅ Integration tests (complete workflows, validation)
✅ Example flows (data processing, ETL, parallel maps)

Migration Guide

No breaking changes. Existing flows continue to work unchanged. To use map steps:

For root maps (array input):
For dependent maps:

Future Considerations

The input.run convention will be removed in future versions
Map steps are ready for this transition (handlers don't use input.run)
Type system properly handles both root and dependent map cases

Checklist

Related Issues

Part of the map infrastructure implementation series (see PLAN.md)

changeset-bot · 2025-09-18T06:26:27Z

🦋 Changeset detected

Latest commit: d52a0e0

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 7 packages

Name	Type
@pgflow/dsl	Patch
@pgflow/client	Patch
@pgflow/core	Patch
@pgflow/edge-worker	Patch
@pgflow/example-flows	Patch
pgflow	Patch
@pgflow/website	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

coderabbitai · 2025-09-18T06:26:30Z

Important

Review skipped

Auto reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

✨ Finishing touches

🧪 Generate unit tests

Create PR with unit tests
Post copyable unit tests in a comment
Commit unit tests in branch 09-18-add-map-support-to-dsl

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

jumski · 2025-09-18T06:26:47Z

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

How to use the Graphite Merge Queue

Add either label to this PR to merge it via the merge queue:

merge:queue - adds this PR to the back of the merge queue
hotfix:queue - for urgent hot fixes, skip the queue and merge this PR next

You must have a Graphite account in order to use the merge queue. Sign up using this link.

_{An organization admin has enabled the Graphite Merge Queue in this repository.} _{Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue.}

This stack of pull requests is managed by Graphite. Learn more about stacking.

nx-cloud · 2025-09-18T06:30:57Z

View your CI Pipeline Execution ↗ for commit d52a0e0

Command	Status	Duration	Result
`nx run-many -t build --projects client,dsl --co...`	✅ Succeeded	4s	View ↗
`nx affected -t build --configuration=production...`	✅ Succeeded	3s	View ↗
`nx affected -t lint typecheck test --parallel -...`	✅ Succeeded	6m 4s	View ↗

☁️ Nx Cloud last updated this comment at 2025-10-06 15:33:35 UTC

…nts, and edge cases - Introduced new integration tests for flow compilation involving map steps - Added runtime validation tests for step dependencies and slug validation - Included type inference validation tests for map method constraints - Covered edge cases such as empty arrays and various runtime options - Added tests for flow with only map steps and multiple independent chains - Ensured correct parameter ordering and handling of dependencies in compiled SQL - Expanded coverage for root and dependent map compilation scenarios - Included tests for flow with only map steps and chaining behaviors - Improved test suite robustness for map step handling in flow compilation

github-actions · 2025-10-06T15:33:30Z

🔍 Preview Deployment: Website

✅ Deployment successful!

🔗 Preview URL: https://pr-218.pgflow.pages.dev

📝 Details:

Branch: 09-18-add-map-support-to-dsl
Commit: ed32a9a3b18e071545a4173f035e6d338620cce0
View Logs

_Last updated: _

github-actions · 2025-10-06T15:35:19Z

🔍 Preview Deployment: Playground

✅ Deployment successful!

🔗 Preview URL: https://pr-218--pgflow-demo.netlify.app

📝 Details:

Branch: 09-18-add-map-support-to-dsl
Commit: ed32a9a3b18e071545a4173f035e6d338620cce0
View Logs

_Last updated: _

graphite-app · 2025-10-07T08:28:21Z

Merge activity

Oct 7, 8:28 AM UTC: jumski added this pull request to the Graphite merge queue.
Oct 7, 8:29 AM UTC: CI is running for this pull request on a draft pull request (#232) due to your merge queue CI optimization settings.
Oct 7, 8:29 AM UTC: Merged by the Graphite merge queue via draft PR: #232.

## Overview This PR introduces the `.map()` method to the Flow DSL, enabling users to define map-type steps that process arrays element-by-element. This completes the DSL support for the map step infrastructure that was previously added to the SQL Core layer. ## Problem Previously, while the SQL Core supported map steps (with `step_type => 'map'`), there was no way to define them via the TypeScript DSL. Users had to manually write SQL or work around this limitation. ## Solution The new `.map()` method provides a type-safe, intuitive way to define map steps with these key characteristics: ### API Design ``` // Root map - processes flow input array new Flow<string[]>({ slug: 'text_processing' }) .map({ slug: 'normalize' }, (text) => text.trim().toLowerCase()) // Dependent map - processes another step's array output new Flow<{}>({ slug: 'workflow' }) .array({ slug: 'fetch_items' }, () => fetchData()) .map({ slug: 'enrich', array: 'fetch_items' }, (item) => ({ ...item, enriched: true })) ``` ### Key Features 1. **Intuitive dependency syntax** - Omit `array` property for root maps (processes flow input) - Use `array: 'stepSlug'` for dependent maps (single dependency only) 2. **Different handler signature** - Map handler: `(item, context) => Json` - Regular step handler: `(input, context) => Json` - Map handlers receive individual array elements, not the full input object 3. **Type safety** - Enforces Json-compatible types for inputs and outputs - Full TypeScript inference for item types based on dependencies - Compile-time validation that dependencies return arrays 4. **SQL generation** - Correctly adds `step_type => 'map'` parameter to `pgflow.add_step()` - Maintains proper parameter ordering: dependencies, options, then step_type ## Implementation Details ### Changes Made 1. **Extended StepDefinition interface** (`dsl.ts`) - Added optional `stepType?: 'single' | 'map'` property - Defaults to 'single' for backward compatibility 2. **Implemented** **`.map()`** **method** (`dsl.ts`) - Complex type signatures for proper inference - Runtime validation for dependencies - Stores `stepType: 'map'` in step definition 3. **Updated compile function** (`compile-flow.ts`) - Checks for `stepType` property - Adds `step_type => 'map'` parameter when needed 4. **Comprehensive test coverage** - Type tests: 20 tests covering constraints and inference - Runtime tests: 12 tests for SQL generation - Integration tests: 12 tests for complete workflows - Example flows demonstrating various use cases ### Technical Decisions - **Optional** **`array`** **key**: Mirrors the `dependsOn` pattern (no key = root step) - **Single dependency constraint**: Enforced at both type and runtime levels - **Handler signature difference**: Clear distinction between map and regular steps - **Type parameter ordering**: Worked around TypeScript constraints for optional parameters ## Testing The implementation follows TDD principles with: - ✅ Type safety tests (inference, constraints, edge cases) - ✅ SQL generation tests (parameter ordering, step_type inclusion) - ✅ Integration tests (complete workflows, validation) - ✅ Example flows (data processing, ETL, parallel maps) ## Migration Guide No breaking changes. Existing flows continue to work unchanged. To use map steps: 1. For root maps (array input): 2. For dependent maps: ## Future Considerations - The `input.run` convention will be removed in future versions - Map steps are ready for this transition (handlers don't use `input.run`) - Type system properly handles both root and dependent map cases ## Checklist - [x] Implementation complete with type safety - [x] SQL generation working correctly - [x] Comprehensive test coverage - [x] Example flows created - [x] No breaking changes - [x] Documentation in changeset ## Related Issues Part of the map infrastructure implementation series (see [PLAN.md](http://PLAN.md))

jumski mentioned this pull request Sep 18, 2025

feat: add Map Step Output Aggregation #217

Closed

jumski changed the title ~~test: add comprehensive tests for map step compilation, type constraints, and edge cases~~ feat: Add .map() Method to Flow DSL for Map Step Support Sep 18, 2025

jumski marked this pull request as ready for review September 18, 2025 06:27

jumski force-pushed the 09-18-add-map-support-to-dsl branch from 07f0d62 to bb3491c Compare September 18, 2025 06:30

jumski force-pushed the 09-18-add-map-support-to-dsl branch from bb3491c to 429b748 Compare September 18, 2025 07:32

jumski force-pushed the 09-17-add-map-step-output-aggregation branch from b77f743 to 2b1f97e Compare September 18, 2025 07:32

jumski mentioned this pull request Sep 18, 2025

chore: improve verify-schemas-synced and regenerate-temp-migration #219

Closed

jumski force-pushed the 09-18-add-map-support-to-dsl branch from 429b748 to 0ed394f Compare September 18, 2025 07:42

jumski force-pushed the 09-17-add-map-step-output-aggregation branch from 2b1f97e to fcb8222 Compare September 18, 2025 07:42

jumski force-pushed the 09-18-add-map-support-to-dsl branch 5 times, most recently from 252a37d to 83609aa Compare September 18, 2025 08:46

jumski force-pushed the 09-17-add-map-step-output-aggregation branch from fcb8222 to 7422f2a Compare September 18, 2025 08:46

jumski force-pushed the 09-18-add-map-support-to-dsl branch 4 times, most recently from 8e2d37b to 18fd9d1 Compare September 18, 2025 10:03

jumski force-pushed the 09-18-add-map-support-to-dsl branch from 18fd9d1 to 53cc331 Compare September 18, 2025 10:08

jumski temporarily deployed to preview September 18, 2025 10:15 — with GitHub Actions Inactive

This was referenced Sep 18, 2025

Fix Orphaned Messages on Run Failure #220

Closed

feat: add task_index attribute to step_task_record and start_tasks function #221

Closed

test: add integration tests for flows with map steps #222

Closed

jumski force-pushed the 09-17-add-map-step-output-aggregation branch from 7422f2a to c1c4c34 Compare October 5, 2025 18:59

jumski force-pushed the 09-18-add-map-support-to-dsl branch from 53cc331 to 43c4799 Compare October 5, 2025 18:59

jumski mentioned this pull request Oct 5, 2025

docs: improve documentation and READMEs to explain map/array steps #228

Closed

jumski force-pushed the 09-17-add-map-step-output-aggregation branch from c1c4c34 to 8ec0268 Compare October 5, 2025 19:20

jumski force-pushed the 09-18-add-map-support-to-dsl branch from 43c4799 to 26104e7 Compare October 5, 2025 19:20

jumski temporarily deployed to preview October 5, 2025 19:28 — with GitHub Actions Inactive

jumski mentioned this pull request Oct 6, 2025

fix: remove temp migrations and prepare stack to be merged #230

Closed

jumski force-pushed the 09-18-add-map-support-to-dsl branch from 26104e7 to d52a0e0 Compare October 6, 2025 15:25

jumski force-pushed the 09-17-add-map-step-output-aggregation branch from 8ec0268 to 02056ce Compare October 6, 2025 15:25

jumski temporarily deployed to preview October 6, 2025 15:32 — with GitHub Actions Inactive

jumski mentioned this pull request Oct 6, 2025

fix: typecheck tests using strict tsc mode #231

Closed

graphite-app bot closed this Oct 7, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: Add `.map()` Method to Flow DSL for Map Step Support #218

feat: Add `.map()` Method to Flow DSL for Map Step Support #218

Uh oh!

jumski commented Sep 18, 2025 •

edited

Loading

Uh oh!

changeset-bot bot commented Sep 18, 2025 •

edited

Loading

Uh oh!

coderabbitai bot commented Sep 18, 2025

Review skipped

Uh oh!

jumski commented Sep 18, 2025 •

edited

Loading

Uh oh!

nx-cloud bot commented Sep 18, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Oct 6, 2025

Uh oh!

github-actions bot commented Oct 6, 2025

Uh oh!

graphite-app bot commented Oct 7, 2025 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

feat: Add .map() Method to Flow DSL for Map Step Support #218

feat: Add .map() Method to Flow DSL for Map Step Support #218

Uh oh!

Conversation

jumski commented Sep 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Problem

Solution

API Design

Key Features

Implementation Details

Changes Made

Technical Decisions

Testing

Migration Guide

Future Considerations

Checklist

Related Issues

Uh oh!

changeset-bot bot commented Sep 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

coderabbitai bot commented Sep 18, 2025

Review skipped

Uh oh!

jumski commented Sep 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

How to use the Graphite Merge Queue

Uh oh!

nx-cloud bot commented Sep 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Oct 6, 2025

🔍 Preview Deployment: Website

Uh oh!

github-actions bot commented Oct 6, 2025

🔍 Preview Deployment: Playground

Uh oh!

graphite-app bot commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge activity

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

feat: Add `.map()` Method to Flow DSL for Map Step Support #218

feat: Add `.map()` Method to Flow DSL for Map Step Support #218

jumski commented Sep 18, 2025 •

edited

Loading

changeset-bot bot commented Sep 18, 2025 •

edited

Loading

jumski commented Sep 18, 2025 •

edited

Loading

nx-cloud bot commented Sep 18, 2025 •

edited

Loading

graphite-app bot commented Oct 7, 2025 •

edited

Loading