Skip to content

Conversation

jumski
Copy link
Contributor

@jumski jumski commented Sep 17, 2025

This PR implements output aggregation for map steps, completing the critical functionality needed for map steps to pass their results to dependent steps.

What this adds:

  • Aggregates outputs from all map step tasks into a single array when the step completes
  • Preserves task_index ordering in the aggregated output array
  • Handles edge cases including empty maps (0 tasks) and NULL outputs from individual tasks
  • Optimizes map-to-map data flow to avoid unnecessary aggregation/decomposition cycles
  • Ensures dependent steps receive properly aggregated arrays from completed map steps

Context:

Map steps spawn N tasks to process array elements in parallel. This PR ensures their outputs are properly combined back into an array for consumption by downstream steps, completing the map
step infrastructure in the SQL core layer.

Copy link

changeset-bot bot commented Sep 17, 2025

⚠️ No Changeset found

Latest commit: 02056ce

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

Copy link
Contributor

coderabbitai bot commented Sep 17, 2025

Important

Review skipped

Auto reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Note

Other AI code review bot(s) detected

CodeRabbit has detected other AI code review bot(s) in this pull request and will avoid duplicating their findings in the review comments. This may lead to a less comprehensive review.

✨ Finishing touches
🧪 Generate unit tests
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch 09-17-add-map-step-output-aggregation

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Contributor Author

jumski commented Sep 17, 2025

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more


How to use the Graphite Merge Queue

Add either label to this PR to merge it via the merge queue:

  • merge:queue - adds this PR to the back of the merge queue
  • hotfix:queue - for urgent hot fixes, skip the queue and merge this PR next

You must have a Graphite account in order to use the merge queue. Sign up using this link.

An organization admin has enabled the Graphite Merge Queue in this repository.

Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue.

This stack of pull requests is managed by Graphite. Learn more about stacking.

Copy link

nx-cloud bot commented Sep 17, 2025

🤖 Nx Cloud AI Fix Eligible

An automatically generated fix could have helped fix failing tasks for this run, but Self-healing CI is disabled for this workspace. Visit workspace settings to enable it and get automatic fixes in future runs.

To disable these notifications, a workspace admin can disable them in workspace settings.


View your CI Pipeline Execution ↗ for commit 02056ce

Command Status Duration Result
nx affected -t lint typecheck test --parallel -... ❌ Failed 5m 56s View ↗

☁️ Nx Cloud last updated this comment at 2025-10-06 15:32:55 UTC

@jumski jumski changed the title chore: update PLAN.md with current feature and task statuses feat: add Map Step Output Aggregation Sep 17, 2025
@jumski jumski marked this pull request as ready for review September 17, 2025 07:12
@jumski jumski force-pushed the 09-17-add-map-step-output-aggregation branch from 1cc39ef to 93cd2af Compare September 17, 2025 10:54
@jumski jumski force-pushed the 09-17-add-map-step-output-aggregation branch from 93cd2af to a61429b Compare September 17, 2025 12:07
@jumski jumski force-pushed the 09-17-add-map-step-output-aggregation branch from a61429b to b01eb63 Compare September 17, 2025 13:10
@jumski jumski force-pushed the 09-17-add-map-step-output-aggregation branch 3 times, most recently from 5a0ef17 to 9ab5b0e Compare September 17, 2025 19:16
@jumski jumski force-pushed the 09-17-add-map-step-output-aggregation branch from b77f743 to 2b1f97e Compare September 18, 2025 07:32
@jumski jumski force-pushed the 09-17-add-map-step-output-aggregation branch 2 times, most recently from fcb8222 to 7422f2a Compare September 18, 2025 08:46
@jumski jumski force-pushed the 09-17-add-map-step-output-aggregation branch from 7422f2a to c1c4c34 Compare October 5, 2025 18:59
@jumski jumski force-pushed the 09-16-handle-arrays-in-start-tasks branch from 781e413 to 53ca34b Compare October 5, 2025 18:59
@jumski jumski force-pushed the 09-17-add-map-step-output-aggregation branch from c1c4c34 to 8ec0268 Compare October 5, 2025 19:20
@jumski jumski force-pushed the 09-16-handle-arrays-in-start-tasks branch from 53ca34b to 99d14b5 Compare October 5, 2025 19:20
Copy link
Contributor

github-actions bot commented Oct 5, 2025

🔍 Preview Deployment: Website

Deployment successful!

🔗 Preview URL: https://pr-217.pgflow.pages.dev

📝 Details:

  • Branch: 09-17-add-map-step-output-aggregation
  • Commit: 7f76309e93ac5c41b2efef1a9037b16a2502fd70
  • View Logs

_Last updated: _

Copy link
Contributor

github-actions bot commented Oct 5, 2025

🔍 Preview Deployment: Playground

Deployment successful!

🔗 Preview URL: https://pr-217--pgflow-demo.netlify.app

📝 Details:

  • Branch: 09-17-add-map-step-output-aggregation
  • Commit: 7f76309e93ac5c41b2efef1a9037b16a2502fd70
  • View Logs

_Last updated: _

Reflects ongoing work on map step output aggregation, related migration, testing,
and documentation efforts, along with current implementation status and pending
tasks for map step support in the project.
@jumski jumski force-pushed the 09-16-handle-arrays-in-start-tasks branch from 99d14b5 to e87df3c Compare October 6, 2025 15:25
@jumski jumski force-pushed the 09-17-add-map-step-output-aggregation branch from 8ec0268 to 02056ce Compare October 6, 2025 15:25
Copy link
Contributor

graphite-app bot commented Oct 7, 2025

Merge activity

  • Oct 7, 8:28 AM UTC: jumski added this pull request to the Graphite merge queue.
  • Oct 7, 8:29 AM UTC: CI is running for this pull request on a draft pull request (#232) due to your merge queue CI optimization settings.
  • Oct 7, 8:29 AM UTC: Merged by the Graphite merge queue via draft PR: #232.

graphite-app bot pushed a commit that referenced this pull request Oct 7, 2025
This PR implements output aggregation for map steps, completing the critical functionality needed for map steps to pass their results to dependent steps.

What this adds:

- Aggregates outputs from all map step tasks into a single array when the step completes
- Preserves task_index ordering in the aggregated output array
- Handles edge cases including empty maps (0 tasks) and NULL outputs from individual tasks
- Optimizes map-to-map data flow to avoid unnecessary aggregation/decomposition cycles
- Ensures dependent steps receive properly aggregated arrays from completed map steps

Context:

Map steps spawn N tasks to process array elements in parallel. This PR ensures their outputs are properly combined back into an array for consumption by downstream steps, completing the map
step infrastructure in the SQL core layer.
@graphite-app graphite-app bot closed this Oct 7, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant