Skip to content

Commit 1cc39ef

Browse files
jumskijumski
committed
chore: update PLAN.md with current feature and task statuses
Reflects ongoing work on map step output aggregation, related migration, testing, and documentation efforts, along with current implementation status and pending tasks for map step support in the project.
1 parent 781e413 commit 1cc39ef

File tree

3 files changed

+332
-2
lines changed

3 files changed

+332
-2
lines changed

‎PERFORMANCE.md‎

Lines changed: 139 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,139 @@
1+
# Performance Measurements - Output Aggregation
2+
3+
## Stage 1: Baseline (Before Output Aggregation)
4+
Date: 2025-01-17
5+
Branch: 09-17-add-map-step-output-aggregation (before changes)
6+
7+
### Test Setup
8+
- Database: PostgreSQL (local Supabase)
9+
- Test data: TBD
10+
- Hardware: Local development machine
11+
12+
### Core Function Performance
13+
14+
#### `start_tasks`
15+
- **Description**: Starts tasks and constructs inputs from dependencies
16+
- **Test scenario**: TBD
17+
- **Average execution time**: TBD ms
18+
- **Min/Max**: TBD ms / TBD ms
19+
- **Notes**: Hot path - called for every ready step
20+
21+
#### `complete_task`
22+
- **Description**: Completes task and updates dependencies
23+
- **Test scenario**: TBD
24+
- **Average execution time**: TBD ms
25+
- **Min/Max**: TBD ms / TBD ms
26+
- **Notes**: Called for every task completion
27+
28+
#### `maybe_complete_run`
29+
- **Description**: Completes run and aggregates leaf outputs
30+
- **Test scenario**: TBD
31+
- **Average execution time**: TBD ms
32+
- **Min/Max**: TBD ms / TBD ms
33+
- **Notes**: Called once per run completion
34+
35+
### Map-Specific Performance
36+
37+
#### Map Task Spawning
38+
- **Test**: Creating N tasks for map step
39+
- **Array sizes tested**: [10, 100, 1000]
40+
- **Results**:
41+
- 10 items: TBD ms
42+
- 100 items: TBD ms
43+
- 1000 items: TBD ms
44+
45+
#### Map Task Completion
46+
- **Test**: Completing all tasks in a map step
47+
- **Array sizes tested**: [10, 100, 1000]
48+
- **Results**:
49+
- 10 items: TBD ms
50+
- 100 items: TBD ms
51+
- 1000 items: TBD ms
52+
53+
---
54+
55+
## Stage 2: Naive Implementation (Inline Aggregation)
56+
Date: TBD
57+
Branch: 09-17-add-map-step-output-aggregation (with inline aggregation)
58+
59+
### Changes Made
60+
- Inline aggregation in `start_tasks` deps CTE
61+
- Inline aggregation in `maybe_complete_run`
62+
- Inline aggregation in `complete_task` for broadcasts
63+
64+
### Core Function Performance
65+
66+
#### `start_tasks`
67+
- **Average execution time**: TBD ms
68+
- **Performance impact**: TBD% (compared to baseline)
69+
- **Notes**: TBD
70+
71+
#### `complete_task`
72+
- **Average execution time**: TBD ms
73+
- **Performance impact**: TBD% (compared to baseline)
74+
- **Notes**: TBD
75+
76+
#### `maybe_complete_run`
77+
- **Average execution time**: TBD ms
78+
- **Performance impact**: TBD% (compared to baseline)
79+
- **Notes**: TBD
80+
81+
---
82+
83+
## Stage 3: Optimized Map-to-Map Transfer
84+
Date: TBD
85+
Branch: 09-17-add-map-step-output-aggregation (with optimization)
86+
87+
### Changes Made
88+
- Direct task-to-task output transfer for map->map dependencies
89+
- Avoids aggregation and decomposition overhead
90+
91+
### Map-to-Map Performance
92+
93+
#### Direct Transfer vs Aggregation
94+
- **Test**: Map(10) -> Map(10) dependency chain
95+
- **Naive approach**: TBD ms
96+
- **Optimized approach**: TBD ms
97+
- **Improvement**: TBD%
98+
99+
#### Scaling Test
100+
- **Test**: Map(N) -> Map(N) with varying sizes
101+
- **Results**:
102+
- 10 items: TBD ms (improvement: TBD%)
103+
- 100 items: TBD ms (improvement: TBD%)
104+
- 1000 items: TBD ms (improvement: TBD%)
105+
106+
---
107+
108+
## Stage 4: Function Extraction (Optional)
109+
Date: TBD
110+
Branch: TBD
111+
112+
### Changes Made
113+
- Extracted aggregation to `pgflow.get_step_output()` helper function
114+
115+
### Function Call Overhead
116+
117+
#### `get_step_output` Performance
118+
- **Average execution time**: TBD ms
119+
- **Overhead compared to inline**: TBD ms (TBD%)
120+
121+
### Overall Impact
122+
- **Total performance impact**: TBD%
123+
- **Recommendation**: TBD (extract/keep inline)
124+
125+
---
126+
127+
## Summary and Recommendations
128+
129+
### Key Findings
130+
- TBD
131+
132+
### Performance Bottlenecks
133+
- TBD
134+
135+
### Recommended Approach
136+
- TBD
137+
138+
### Future Optimizations
139+
- TBD

‎PLAN.md‎

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,13 +2,21 @@
22

33
**NOTE: This PLAN.md file should be removed in the final PR once all map infrastructure is complete.**
44

5-
### Current State
5+
### Features
66

77
- ✅ **WORKING**: Empty array maps (taskless) cascade and complete correctly
88
- ✅ **WORKING**: Task spawning creates N tasks with correct indices
99
- ✅ **WORKING**: Dependency count propagation for map steps
1010
- ✅ **WORKING**: Array element extraction - tasks get full array instead of individual items
11-
- ❌ **MISSING**: Output aggregation - no way to combine map task outputs for dependents
11+
- 🛠️ **CURRENT**: Output aggregation - no way to combine map task outputs for dependents
12+
- ⏳ **WAITING**: DSL support for `.map()` for defining map steps
13+
14+
### Chores
15+
16+
- ⏳ **WAITING**: Integration tests for map steps
17+
- ⏳ **WAITING**: Consolidated migration for map steps
18+
- ⏳ **WAITING**: Documentation for map steps
19+
- ⏳ **WAITING**: Graphite stack merge for map steps
1220

1321
## Implementation Status
1422

‎PLAN_output_aggregation.md‎

Lines changed: 183 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,183 @@
1+
# Output Aggregation Implementation Plan
2+
3+
## Overview
4+
Implement output aggregation for map steps with performance-focused, test-first approach.
5+
6+
## Stage 1: Baseline Performance Measurement
7+
8+
### Tasks
9+
- Run existing performance tests multiple times (3-5 runs)
10+
- Calculate average values for each metric
11+
- Document results in `PERFORMANCE.md`
12+
13+
### Commands
14+
```bash
15+
# Run performance tests (repeat 3-5 times)
16+
pnpm nx test:pgtap core -- pkgs/core/tests/performance/*.sql
17+
18+
# Document results in PERFORMANCE.md with format:
19+
# - Test name
20+
# - Average execution time
21+
# - Min/Max values
22+
# - Standard deviation if significant
23+
```
24+
25+
## Stage 2: Test-First Development (Naive Implementation)
26+
27+
### Approach
28+
Write failing tests one at a time, implement inline solution to make them pass.
29+
30+
### Test Scenarios (in order of complexity)
31+
1. **Basic map output aggregation**
32+
- Single map step with 3 tasks
33+
- Verify outputs aggregated in task_index order
34+
35+
2. **Empty map output**
36+
- Map step with 0 tasks
37+
- Should return `[]` as output
38+
39+
3. **Map feeding into single step**
40+
- Map step output aggregated as array
41+
- Single step receives full array as dependency input
42+
43+
4. **Map feeding into another map**
44+
- First map outputs array
45+
- Second map processes each element
46+
47+
5. **Edge case: NULL outputs**
48+
- Some tasks return NULL
49+
- Aggregation should include NULLs in array
50+
51+
6. **Run completion with map leaf step**
52+
- Map step as leaf (no dependents)
53+
- Run output should contain aggregated array
54+
55+
### Development Workflow
56+
```bash
57+
# 1. Write test
58+
vim pkgs/core/tests/map_output_aggregation_test.sql
59+
60+
# 2. Run test (should fail)
61+
pkgs/core/scripts/run-test-with-colors pkgs/core/tests/map_output_aggregation_test.sql
62+
63+
# 3. Update functions in database
64+
psql $DATABASE_URL -f updated_function.sql
65+
66+
# 4. Re-run test (iterate until passing)
67+
pkgs/core/scripts/run-test-with-colors pkgs/core/tests/map_output_aggregation_test.sql
68+
69+
# 5. Repeat for next test scenario
70+
```
71+
72+
### Implementation Notes
73+
**Naive approach**: Inline aggregation directly in the affected functions
74+
- **`start_tasks`**: Aggregate map outputs inline in deps CTE
75+
- **`maybe_complete_run`**: Aggregate map outputs for leaf steps
76+
- **`complete_task`**: Aggregate for broadcast events
77+
78+
## Stage 3: Performance Measurement (Naive)
79+
80+
### Tasks
81+
- Run performance tests with naive implementation
82+
- Compare with baseline
83+
- Document in `PERFORMANCE.md`
84+
85+
### Expected Impact
86+
- `start_tasks`: Moderate overhead (aggregation per dependency)
87+
- `maybe_complete_run`: Minimal (only at run completion)
88+
- `complete_task`: Minimal (only for broadcasts)
89+
90+
## Stage 4: Map-to-Map Optimization
91+
92+
### Concept
93+
Optimize the map->map case where we aggregate outputs only to immediately decompose them:
94+
- Map A task[i] → output[i]
95+
- Currently: Aggregate to array → decompose in Map B
96+
- Optimized: Map A task[i] → Map B task[i] directly
97+
98+
### Implementation Strategy
99+
```sql
100+
-- In start_tasks deps CTE, add special case:
101+
CASE
102+
WHEN step.step_type = 'map' AND dep_step.step_type = 'map' THEN
103+
-- Direct task-to-task transfer
104+
(SELECT output FROM pgflow.step_tasks
105+
WHERE run_id = st.run_id
106+
AND step_slug = dep.dep_slug
107+
AND task_index = st.task_index
108+
AND status = 'completed')
109+
ELSE
110+
-- Standard aggregation for non-map dependents
111+
...
112+
END
113+
```
114+
115+
### Tests
116+
1. **Map-to-map direct transfer**
117+
- Verify task[i] gets output[i] without aggregation
118+
119+
2. **Map-to-map with different sizes**
120+
- Source map: 5 tasks
121+
- Target map: 5 tasks (should work)
122+
- Error handling if sizes mismatch
123+
124+
## Stage 5: Final Performance Measurement
125+
126+
### Tasks
127+
- Run all performance tests
128+
- Compare baseline vs naive vs optimized
129+
- Document final results and recommendations
130+
131+
### Metrics to Track
132+
- Execution time per function
133+
- Memory usage (if measurable)
134+
- Query complexity (EXPLAIN ANALYZE)
135+
136+
## Stage 6: Function Extraction Decision
137+
138+
### Evaluation Criteria
139+
After measuring performance of inline implementation:
140+
1. **Performance overhead**: Is function call cost acceptable?
141+
2. **Code duplication**: How much repetition exists?
142+
3. **Maintainability**: Would function improve code clarity?
143+
144+
### If extracting to function:
145+
```sql
146+
-- Create pgflow.get_step_output() helper
147+
-- Update all three locations to use helper
148+
-- Re-run performance tests
149+
-- Document final decision and rationale
150+
```
151+
152+
## Notes for Implementation
153+
154+
### Key Files to Modify
155+
1. `pkgs/core/schemas/0120_function_start_tasks.sql` (lines 46-53)
156+
2. `pkgs/core/schemas/0100_function_maybe_complete_run.sql` (lines 16-27)
157+
3. `pkgs/core/schemas/0100_function_complete_task.sql` (line 156)
158+
159+
### Testing Database Access
160+
```bash
161+
# Get database URL
162+
source .env.local
163+
echo $DATABASE_URL
164+
165+
# Direct psql access for function updates
166+
psql $DATABASE_URL
167+
168+
# View current function
169+
\sf pgflow.start_tasks
170+
```
171+
172+
### Performance Testing Tips
173+
- Run tests when system is idle
174+
- Use consistent hardware/environment
175+
- Warm up database before measurements
176+
- Consider connection pooling effects
177+
178+
## Success Criteria
179+
- [ ] All map output aggregation tests passing
180+
- [ ] Performance impact < 10% for typical workflows
181+
- [ ] Map-to-map optimization shows measurable improvement
182+
- [ ] Documentation complete with performance analysis
183+
- [ ] Decision made on function extraction based on data

0 commit comments

Comments
 (0)