feat: track template change classification in execution_start telemetry#12309
Draft
christian-byrne wants to merge 3 commits into
Draft
feat: track template change classification in execution_start telemetry#12309christian-byrne wants to merge 3 commits into
christian-byrne wants to merge 3 commits into
Conversation
Adds template_change_type property to the PostHog execution_start event so analytics can distinguish runs that explore variations (seed/prompt only) from runs that materially change the workflow graph. On template load, the original workflow JSON is captured as a baseline. At execution time, the baseline is diffed against the current workflow: - unchanged: no diff - seed_only: only seed-named widget values differ - prompt_only: only prompt/text-named widget values differ - seed_and_prompt: both seed and prompt widget values differ - structural: any other diff (node added/removed, link change, type change, non-seed/non-prompt widget value, length mismatch) The field is only attached when is_template is true and a baseline was captured; it is omitted for custom workflows and templates loaded before this change.
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughAdds baseline storage, a classifier that labels template edits as ChangesTemplate workflow change tracking
sequenceDiagram
participant User
participant loadWorkflowTemplate
participant templateBaselineStore
participant getExecutionContext
participant classifyTemplateChange
User->>loadWorkflowTemplate: Load template
loadWorkflowTemplate->>templateBaselineStore: setTemplateBaseline(id, baseline)
templateBaselineStore->>templateBaselineStore: Deep-clone & store
User->>getExecutionContext: Collect execution telemetry
getExecutionContext->>templateBaselineStore: getTemplateBaseline(id)
getExecutionContext->>classifyTemplateChange: classifyTemplateChange(baseline, current, liveNodes)
classifyTemplateChange->>classifyTemplateChange: Compare definitions, nodes, links, widgets
classifyTemplateChange-->>getExecutionContext: template_change_type
getExecutionContext-->>User: ExecutionContext with classification
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Suggested labels
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 6 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (6 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
🎨 Storybook: ✅ Built — View Storybook |
🎭 Playwright: ✅ 1604 passed, 0 failed · 2 flaky📊 Browser Reports
|
📦 Bundle: 5.36 MB gzip 🔴 +219 BDetailsSummary
Category Glance App Entry Points — 26.1 kB (baseline 26.1 kB) • ⚪ 0 BMain entry bundles and manifests
Status: 1 added / 1 removed Graph Workspace — 1.24 MB (baseline 1.24 MB) • ⚪ 0 BGraph editor runtime, canvas, workflow orchestration
Status: 1 added / 1 removed Views & Navigation — 82.9 kB (baseline 82.9 kB) • ⚪ 0 BTop-level views, pages, and routed surfaces
Status: 9 added / 9 removed / 2 unchanged Panels & Settings — 527 kB (baseline 527 kB) • ⚪ 0 BConfiguration panels, inspectors, and settings screens
Status: 10 added / 10 removed / 14 unchanged User & Accounts — 17.8 kB (baseline 17.8 kB) • ⚪ 0 BAuthentication, profile, and account management bundles
Status: 5 added / 5 removed / 2 unchanged Editors & Dialogs — 112 kB (baseline 112 kB) • ⚪ 0 BModals, dialogs, drawers, and in-app editors
Status: 4 added / 4 removed UI Components — 58 kB (baseline 58 kB) • ⚪ 0 BReusable component library chunks
Status: 5 added / 5 removed / 8 unchanged Data & Services — 3.16 MB (baseline 3.16 MB) • 🔴 +758 BStores, services, APIs, and repositories
Status: 13 added / 13 removed / 4 unchanged Utilities & Hooks — 366 kB (baseline 366 kB) • ⚪ 0 BHelpers, composables, and utility bundles
Status: 13 added / 13 removed / 18 unchanged Vendor & Third-Party — 9.94 MB (baseline 9.94 MB) • ⚪ 0 BExternal libraries and shared vendor chunks Status: 16 unchanged Other — 9.16 MB (baseline 9.16 MB) • ⚪ 0 BBundles that do not match a named category
Status: 57 added / 57 removed / 86 unchanged ⚡ Performance Report
No regressions detected. All metrics
Historical variance (last 15 runs)
Trend (last 15 commits on main)
Raw data{
"timestamp": "2026-05-16T03:40:56.456Z",
"gitSha": "d72ff5b4b8f329b85a3823716d94c20b09894977",
"branch": "glary/posthog-template-change-classification",
"measurements": [
{
"name": "canvas-idle",
"durationMs": 2047.111000000001,
"styleRecalcs": 9,
"styleRecalcDurationMs": 8.193999999999999,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 385.54299999999995,
"heapDeltaBytes": -1437200,
"heapUsedBytes": 49244640,
"domNodes": 18,
"jsHeapTotalBytes": 17301504,
"scriptDurationMs": 16.906,
"eventListeners": 4,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.66333333333335,
"p95FrameDurationMs": 16.799999999999272
},
{
"name": "canvas-idle",
"durationMs": 2016.9399999999769,
"styleRecalcs": 9,
"styleRecalcDurationMs": 8.613000000000001,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 344.97200000000004,
"heapDeltaBytes": 23176816,
"heapUsedBytes": 71776128,
"domNodes": 18,
"jsHeapTotalBytes": 14417920,
"scriptDurationMs": 15.6,
"eventListeners": 4,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.666666666666668,
"p95FrameDurationMs": 16.799999999999272
},
{
"name": "canvas-mouse-sweep",
"durationMs": 1819.7789999999827,
"styleRecalcs": 73,
"styleRecalcDurationMs": 35.982,
"layouts": 12,
"layoutDurationMs": 3.666,
"taskDurationMs": 787.854,
"heapDeltaBytes": 13888076,
"heapUsedBytes": 62972532,
"domNodes": -263,
"jsHeapTotalBytes": 22147072,
"scriptDurationMs": 121.659,
"eventListeners": -133,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.66333333333335,
"p95FrameDurationMs": 16.800000000000182
},
{
"name": "canvas-mouse-sweep",
"durationMs": 2050.339000000008,
"styleRecalcs": 81,
"styleRecalcDurationMs": 43.117000000000004,
"layouts": 12,
"layoutDurationMs": 3.849,
"taskDurationMs": 1035.339,
"heapDeltaBytes": 5078984,
"heapUsedBytes": 53126936,
"domNodes": -257,
"jsHeapTotalBytes": 14544896,
"scriptDurationMs": 138.871,
"eventListeners": -131,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.666666666666668,
"p95FrameDurationMs": 16.700000000000728
},
{
"name": "canvas-zoom-sweep",
"durationMs": 1718.3239999999955,
"styleRecalcs": 30,
"styleRecalcDurationMs": 15.076,
"layouts": 6,
"layoutDurationMs": 0.5760000000000001,
"taskDurationMs": 279.964,
"heapDeltaBytes": 524628,
"heapUsedBytes": 48910252,
"domNodes": 75,
"jsHeapTotalBytes": 14942208,
"scriptDurationMs": 16.889,
"eventListeners": 19,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.666666666666668,
"p95FrameDurationMs": 16.700000000000728
},
{
"name": "canvas-zoom-sweep",
"durationMs": 1723.8570000000095,
"styleRecalcs": 31,
"styleRecalcDurationMs": 15.976999999999999,
"layouts": 6,
"layoutDurationMs": 0.521,
"taskDurationMs": 307.368,
"heapDeltaBytes": 1041740,
"heapUsedBytes": 53744760,
"domNodes": 78,
"jsHeapTotalBytes": 25165824,
"scriptDurationMs": 24.969000000000005,
"eventListeners": 19,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.666666666666668,
"p95FrameDurationMs": 16.800000000000182
},
{
"name": "dom-widget-clipping",
"durationMs": 561.1769999999865,
"styleRecalcs": 13,
"styleRecalcDurationMs": 8.616,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 352.645,
"heapDeltaBytes": -4222764,
"heapUsedBytes": 63889508,
"domNodes": 22,
"jsHeapTotalBytes": 21233664,
"scriptDurationMs": 62.223,
"eventListeners": 2,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.66333333333332,
"p95FrameDurationMs": 16.799999999999727
},
{
"name": "dom-widget-clipping",
"durationMs": 555.2559999999858,
"styleRecalcs": 11,
"styleRecalcDurationMs": 7.802,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 344.668,
"heapDeltaBytes": 15104744,
"heapUsedBytes": 64086572,
"domNodes": 18,
"jsHeapTotalBytes": 16515072,
"scriptDurationMs": 59.635000000000005,
"eventListeners": 0,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.663333333333338,
"p95FrameDurationMs": 16.700000000000273
},
{
"name": "large-graph-idle",
"durationMs": 2027.8559999999857,
"styleRecalcs": 9,
"styleRecalcDurationMs": 11.927,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 529.696,
"heapDeltaBytes": 19445292,
"heapUsedBytes": 77315152,
"domNodes": -264,
"jsHeapTotalBytes": 5533696,
"scriptDurationMs": 83.673,
"eventListeners": -159,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.66333333333332,
"p95FrameDurationMs": 16.700000000000728
},
{
"name": "large-graph-idle",
"durationMs": 2021.6819999999984,
"styleRecalcs": 8,
"styleRecalcDurationMs": 8.213999999999999,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 584.986,
"heapDeltaBytes": 9516484,
"heapUsedBytes": 68080588,
"domNodes": -264,
"jsHeapTotalBytes": 290816,
"scriptDurationMs": 98.48,
"eventListeners": -129,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.66333333333332,
"p95FrameDurationMs": 16.799999999999272
},
{
"name": "large-graph-pan",
"durationMs": 2119.2680000000055,
"styleRecalcs": 69,
"styleRecalcDurationMs": 17.410999999999998,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 1127.6029999999998,
"heapDeltaBytes": 9194840,
"heapUsedBytes": 68668400,
"domNodes": -262,
"jsHeapTotalBytes": 757760,
"scriptDurationMs": 442.11,
"eventListeners": -129,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.666666666666668,
"p95FrameDurationMs": 16.800000000000182
},
{
"name": "large-graph-pan",
"durationMs": 2171.4739999999892,
"styleRecalcs": 70,
"styleRecalcDurationMs": 20.044999999999998,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 1149.066,
"heapDeltaBytes": -1238796,
"heapUsedBytes": 65027252,
"domNodes": -262,
"jsHeapTotalBytes": 4542464,
"scriptDurationMs": 408.60600000000005,
"eventListeners": -129,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.666666666666668,
"p95FrameDurationMs": 16.700000000000728
},
{
"name": "large-graph-zoom",
"durationMs": 3211.8859999999927,
"styleRecalcs": 67,
"styleRecalcDurationMs": 20.133,
"layouts": 60,
"layoutDurationMs": 7.183999999999999,
"taskDurationMs": 1352.9300000000003,
"heapDeltaBytes": 33130612,
"heapUsedBytes": 100611552,
"domNodes": -262,
"jsHeapTotalBytes": 27668480,
"scriptDurationMs": 481.925,
"eventListeners": -127,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.666666666666668,
"p95FrameDurationMs": 16.700000000000728
},
{
"name": "large-graph-zoom",
"durationMs": 3166.3030000000276,
"styleRecalcs": 65,
"styleRecalcDurationMs": 18.307,
"layouts": 60,
"layoutDurationMs": 7.459,
"taskDurationMs": 1360.026,
"heapDeltaBytes": 14209164,
"heapUsedBytes": 76296900,
"domNodes": -266,
"jsHeapTotalBytes": 4018176,
"scriptDurationMs": 512.412,
"eventListeners": -127,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.666666666666668,
"p95FrameDurationMs": 16.700000000000728
},
{
"name": "minimap-idle",
"durationMs": 2021.2280000000078,
"styleRecalcs": 9,
"styleRecalcDurationMs": 9.454999999999998,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 533.8119999999999,
"heapDeltaBytes": 17722948,
"heapUsedBytes": 76913440,
"domNodes": -263,
"jsHeapTotalBytes": 4222976,
"scriptDurationMs": 87.42999999999999,
"eventListeners": -161,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.666666666666668,
"p95FrameDurationMs": 16.700000000000728
},
{
"name": "minimap-idle",
"durationMs": 2002.1520000000237,
"styleRecalcs": 10,
"styleRecalcDurationMs": 9.666999999999998,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 579.04,
"heapDeltaBytes": 50747612,
"heapUsedBytes": 107885636,
"domNodes": 20,
"jsHeapTotalBytes": 55107584,
"scriptDurationMs": 100.187,
"eventListeners": 4,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.66333333333335,
"p95FrameDurationMs": 16.699999999999818
},
{
"name": "subgraph-dom-widget-clipping",
"durationMs": 549.6570000000247,
"styleRecalcs": 44,
"styleRecalcDurationMs": 8.159,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 346.09399999999994,
"heapDeltaBytes": 9258184,
"heapUsedBytes": 57861444,
"domNodes": 13,
"jsHeapTotalBytes": 15204352,
"scriptDurationMs": 121.23900000000002,
"eventListeners": 8,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.666666666666668,
"p95FrameDurationMs": 16.700000000000273
},
{
"name": "subgraph-dom-widget-clipping",
"durationMs": 575.6539999999859,
"styleRecalcs": 46,
"styleRecalcDurationMs": 10.82,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 373.36499999999995,
"heapDeltaBytes": 15408720,
"heapUsedBytes": 64594020,
"domNodes": 17,
"jsHeapTotalBytes": 16777216,
"scriptDurationMs": 123.407,
"eventListeners": 6,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.66333333333332,
"p95FrameDurationMs": 16.800000000000182
},
{
"name": "subgraph-idle",
"durationMs": 2029.5969999999954,
"styleRecalcs": 10,
"styleRecalcDurationMs": 8.785000000000002,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 405.81899999999996,
"heapDeltaBytes": 19809772,
"heapUsedBytes": 68594068,
"domNodes": -260,
"jsHeapTotalBytes": 17690624,
"scriptDurationMs": 17.459,
"eventListeners": -129,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.666666666666668,
"p95FrameDurationMs": 16.700000000000728
},
{
"name": "subgraph-idle",
"durationMs": 2014.6599999999921,
"styleRecalcs": 10,
"styleRecalcDurationMs": 9.254999999999999,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 348.66,
"heapDeltaBytes": 23878528,
"heapUsedBytes": 73301204,
"domNodes": 20,
"jsHeapTotalBytes": 14680064,
"scriptDurationMs": 19.746,
"eventListeners": 4,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.66333333333332,
"p95FrameDurationMs": 16.700000000000728
},
{
"name": "subgraph-mouse-sweep",
"durationMs": 1712.4050000000182,
"styleRecalcs": 78,
"styleRecalcDurationMs": 37.682,
"layouts": 16,
"layoutDurationMs": 4.341,
"taskDurationMs": 745.4670000000001,
"heapDeltaBytes": 636700,
"heapUsedBytes": 50760460,
"domNodes": -257,
"jsHeapTotalBytes": 20930560,
"scriptDurationMs": 95.735,
"eventListeners": -129,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.66333333333335,
"p95FrameDurationMs": 16.700000000000728
},
{
"name": "subgraph-mouse-sweep",
"durationMs": 1990.8790000000067,
"styleRecalcs": 87,
"styleRecalcDurationMs": 47.297,
"layouts": 16,
"layoutDurationMs": 4.657,
"taskDurationMs": 938.3290000000001,
"heapDeltaBytes": -6976548,
"heapUsedBytes": 59376304,
"domNodes": 71,
"jsHeapTotalBytes": 20279296,
"scriptDurationMs": 101.591,
"eventListeners": 4,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.666666666666668,
"p95FrameDurationMs": 16.700000000000728
},
{
"name": "subgraph-transition-enter",
"durationMs": 932.9500000000053,
"styleRecalcs": 16,
"styleRecalcDurationMs": 27.326999999999998,
"layouts": 4,
"layoutDurationMs": 12.847999999999999,
"taskDurationMs": 724.3899999999999,
"heapDeltaBytes": 30203704,
"heapUsedBytes": 95802552,
"domNodes": 13513,
"jsHeapTotalBytes": 16777216,
"scriptDurationMs": 26.972999999999995,
"eventListeners": 2527,
"totalBlockingTimeMs": 144,
"frameDurationMs": 16.66333333333332,
"p95FrameDurationMs": 16.700000000000728
},
{
"name": "viewport-pan-sweep",
"durationMs": 8166.552999999965,
"styleRecalcs": 249,
"styleRecalcDurationMs": 49.77400000000001,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 3939.7709999999997,
"heapDeltaBytes": 24142692,
"heapUsedBytes": 82996004,
"domNodes": -263,
"jsHeapTotalBytes": 7049216,
"scriptDurationMs": 1457.7099999999998,
"eventListeners": -113,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.66333333333338,
"p95FrameDurationMs": 16.700000000000728
},
{
"name": "viewport-pan-sweep",
"durationMs": 8337.202999999989,
"styleRecalcs": 249,
"styleRecalcDurationMs": 51.678000000000004,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 4169.896000000001,
"heapDeltaBytes": 90011788,
"heapUsedBytes": 148766640,
"domNodes": -264,
"jsHeapTotalBytes": 68390912,
"scriptDurationMs": 1295.915,
"eventListeners": -125,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.666666666666668,
"p95FrameDurationMs": 16.700000000000728
},
{
"name": "vue-large-graph-idle",
"durationMs": 11835.37100000001,
"styleRecalcs": 0,
"styleRecalcDurationMs": 0,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 11823.41,
"heapDeltaBytes": -39188456,
"heapUsedBytes": 173222692,
"domNodes": -8343,
"jsHeapTotalBytes": 27324416,
"scriptDurationMs": 568.31,
"eventListeners": -16464,
"totalBlockingTimeMs": 0,
"frameDurationMs": 17.223333333333358,
"p95FrameDurationMs": 16.700000000000728
},
{
"name": "vue-large-graph-idle",
"durationMs": 13001.764000000037,
"styleRecalcs": 0,
"styleRecalcDurationMs": 0,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 12965.690000000002,
"heapDeltaBytes": -34825920,
"heapUsedBytes": 171842716,
"domNodes": -8331,
"jsHeapTotalBytes": 23916544,
"scriptDurationMs": 656.2570000000001,
"eventListeners": -16461,
"totalBlockingTimeMs": 0,
"frameDurationMs": 18.330000000000048,
"p95FrameDurationMs": 16.799999999999272
},
{
"name": "vue-large-graph-pan",
"durationMs": 14539.017999999998,
"styleRecalcs": 68,
"styleRecalcDurationMs": 19.071000000000005,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 14510.517999999998,
"heapDeltaBytes": -36822784,
"heapUsedBytes": 174427512,
"domNodes": -8331,
"jsHeapTotalBytes": -1859584,
"scriptDurationMs": 889.5470000000001,
"eventListeners": -16488,
"totalBlockingTimeMs": 95,
"frameDurationMs": 17.223333333333237,
"p95FrameDurationMs": 16.799999999999272
},
{
"name": "vue-large-graph-pan",
"durationMs": 15080.689000000006,
"styleRecalcs": 72,
"styleRecalcDurationMs": 19.28100000000005,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 15047.390000000001,
"heapDeltaBytes": -63504792,
"heapUsedBytes": 152011100,
"domNodes": -8337,
"jsHeapTotalBytes": -1511424,
"scriptDurationMs": 995.9340000000001,
"eventListeners": -16486,
"totalBlockingTimeMs": 0,
"frameDurationMs": 17.220000000000073,
"p95FrameDurationMs": 16.799999999999272
},
{
"name": "workflow-execution",
"durationMs": 115.56300000000874,
"styleRecalcs": 12,
"styleRecalcDurationMs": 18.097999999999995,
"layouts": 7,
"layoutDurationMs": 1.8079999999999996,
"taskDurationMs": 93.10499999999999,
"heapDeltaBytes": 3594940,
"heapUsedBytes": 53388900,
"domNodes": 174,
"jsHeapTotalBytes": 0,
"scriptDurationMs": 18.596,
"eventListeners": 21,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.670000000000012,
"p95FrameDurationMs": 16.700000000000273
},
{
"name": "workflow-execution",
"durationMs": 501.38800000001993,
"styleRecalcs": 14,
"styleRecalcDurationMs": 25.843999999999998,
"layouts": 5,
"layoutDurationMs": 1.323,
"taskDurationMs": 161.46500000000003,
"heapDeltaBytes": -1224596,
"heapUsedBytes": 49073876,
"domNodes": -131,
"jsHeapTotalBytes": -135168,
"scriptDurationMs": 26.715000000000003,
"eventListeners": -64,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.66333333333332,
"p95FrameDurationMs": 16.700000000000273
}
]
} |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
src/platform/telemetry/utils/templateBaselineStore.test.ts (1)
22-64: ⚡ Quick winAdd cap/eviction tests for the 32-baseline limit.
The suite doesn’t cover the
MAX_BASELINESeviction branch, so regressions in bounded-store behavior can slip through.As per coding guidelines, "Write tests for all changes, especially bug fixes to catch future regressions."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/platform/telemetry/utils/templateBaselineStore.test.ts` around lines 22 - 64, Add tests exercising the MAX_BASELINES eviction logic: use clearTemplateBaselines(), then repeatedly call setTemplateBaseline(name, makeWorkflow(i)) for i from 1 to MAX_BASELINES+1 (or 33) to confirm the store caps entries at MAX_BASELINES and evicts the oldest entry; assert getTemplateBaseline for the first inserted name is undefined and for the last MAX_BASELINES names returns the expected workflows (check nodes[0].widgets_values to match the inserted index). Also add a test verifying that reinserting an existing name moves/updates it correctly if the store uses LRU eviction (setTemplateBaseline on an existing key then add entries to force eviction and assert the reinserted key is retained). Ensure tests import/ use MAX_BASELINES, setTemplateBaseline, getTemplateBaseline, clearTemplateBaselines, and makeWorkflow to build expectations.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/platform/telemetry/utils/templateBaselineStore.ts`:
- Around line 28-32: getTemplateBaseline currently returns the internal object
reference from baselineByWorkflowName, allowing callers to mutate the stored
baseline; change getTemplateBaseline to return a defensive clone (e.g., use
structuredClone(baseline) or a deep clone utility like
cloneDeep/JSON.parse(JSON.stringify(baseline)) as a fallback) before returning
so callers cannot mutate the internal ComfyWorkflowJSON; keep the function name
getTemplateBaseline and the baselineByWorkflowName lookup, but wrap the
retrieved value in the clone operation and return undefined if no entry exists.
---
Nitpick comments:
In `@src/platform/telemetry/utils/templateBaselineStore.test.ts`:
- Around line 22-64: Add tests exercising the MAX_BASELINES eviction logic: use
clearTemplateBaselines(), then repeatedly call setTemplateBaseline(name,
makeWorkflow(i)) for i from 1 to MAX_BASELINES+1 (or 33) to confirm the store
caps entries at MAX_BASELINES and evicts the oldest entry; assert
getTemplateBaseline for the first inserted name is undefined and for the last
MAX_BASELINES names returns the expected workflows (check
nodes[0].widgets_values to match the inserted index). Also add a test verifying
that reinserting an existing name moves/updates it correctly if the store uses
LRU eviction (setTemplateBaseline on an existing key then add entries to force
eviction and assert the reinserted key is retained). Ensure tests import/ use
MAX_BASELINES, setTemplateBaseline, getTemplateBaseline, clearTemplateBaselines,
and makeWorkflow to build expectations.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: c076efcb-258f-458a-a4f7-16e802862daa
📒 Files selected for processing (8)
src/platform/telemetry/types.tssrc/platform/telemetry/utils/classifyTemplateChange.test.tssrc/platform/telemetry/utils/classifyTemplateChange.tssrc/platform/telemetry/utils/getExecutionContext.test.tssrc/platform/telemetry/utils/getExecutionContext.tssrc/platform/telemetry/utils/templateBaselineStore.test.tssrc/platform/telemetry/utils/templateBaselineStore.tssrc/platform/workflow/templates/composables/useTemplateWorkflows.ts
Codecov Report❌ Patch coverage is
@@ Coverage Diff @@
## main #12309 +/- ##
===========================================
- Coverage 73.79% 59.61% -14.19%
===========================================
Files 1521 1414 -107
Lines 86757 71985 -14772
Branches 23948 19990 -3958
===========================================
- Hits 64023 42913 -21110
- Misses 21905 28599 +6694
+ Partials 829 473 -356
Flags with carried forward coverage won't be shown. Click here to find out more.
... and 1002 files with indirect coverage changes 🚀 New features to boost your workflow:
|
- classifier now normalizes both v0.4 tuple and v1 object link shapes - subgraph definition diffs are treated as structural (conservative) - baseline is captured after loadGraphData() so it matches the normalized state the user actually starts editing from - getTemplateBaseline returns a defensive deep clone - baseline store switched to LRU semantics (re-set refreshes recency) - added v1-link, subgraph-definition, and MAX_BASELINES eviction tests - mocked workflowStore + baselineStore in useTemplateWorkflows.test.ts
|
Addressed the review feedback in 0647d7e — and a self-review by Oracle surfaced three further issues that I fixed in the same commit: CodeRabbit feedback
Oracle self-review (3 additional findings, all fixed)
Verification
Net change: +6 new tests (3 classifier, 3 baseline store), +1 test mock for |
Contributor
There was a problem hiding this comment.
🧹 Nitpick comments (1)
src/platform/workflow/templates/composables/useTemplateWorkflows.test.ts (1)
52-60: ⚡ Quick winAssert baseline-capture behavior in template-load tests.
You added a
setTemplateBaselinemock but no test currently verifies it was called with the expected baseline, so this new behavior can regress silently.Suggested minimal test assertion update
+import { setTemplateBaseline } from '`@/platform/telemetry/utils/templateBaselineStore`' ... it('should load a template from a regular category', async () => { const { loadWorkflowTemplate } = useTemplateWorkflows() mockWorkflowTemplatesStore.isLoaded = true const result = await loadWorkflowTemplate('template1', 'default') await flushPromises() expect(result).toBe(true) expect(fetch).toHaveBeenCalledWith('mock-file-url/templates/template1.json') + expect(vi.mocked(setTemplateBaseline)).toHaveBeenCalledWith('template1', { + nodes: [], + links: [] + }) })Also add one case where
activeWorkflow?.changeTracker?.activeStateis unavailable to verify fallback to fetchedjson.As per coding guidelines: "Write tests for all changes, especially bug fixes to catch future regressions" and "Do not write tests that just test the mocks; ensure tests fail when code behaves unexpectedly."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/platform/workflow/templates/composables/useTemplateWorkflows.test.ts` around lines 52 - 60, Add assertions to the template-load tests to verify the baseline-capture behavior by asserting that the mocked setTemplateBaseline is called with the expected baseline: when useWorkflowStore().activeWorkflow.changeTracker.activeState exists ensure setTemplateBaseline receives that activeState (nodes/links), and add a separate test case where activeWorkflow?.changeTracker?.activeState is undefined to assert the code falls back to the fetched JSON and calls setTemplateBaseline with the fetched json baseline; reference the mocked setTemplateBaseline and useWorkflowStore symbols to locate where to add these assertions and create the missing fallback test.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@src/platform/workflow/templates/composables/useTemplateWorkflows.test.ts`:
- Around line 52-60: Add assertions to the template-load tests to verify the
baseline-capture behavior by asserting that the mocked setTemplateBaseline is
called with the expected baseline: when
useWorkflowStore().activeWorkflow.changeTracker.activeState exists ensure
setTemplateBaseline receives that activeState (nodes/links), and add a separate
test case where activeWorkflow?.changeTracker?.activeState is undefined to
assert the code falls back to the fetched JSON and calls setTemplateBaseline
with the fetched json baseline; reference the mocked setTemplateBaseline and
useWorkflowStore symbols to locate where to add these assertions and create the
missing fallback test.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: d1a8c234-cda8-4a59-bce0-703e57ca8b3b
📒 Files selected for processing (6)
src/platform/telemetry/utils/classifyTemplateChange.test.tssrc/platform/telemetry/utils/classifyTemplateChange.tssrc/platform/telemetry/utils/templateBaselineStore.test.tssrc/platform/telemetry/utils/templateBaselineStore.tssrc/platform/workflow/templates/composables/useTemplateWorkflows.test.tssrc/platform/workflow/templates/composables/useTemplateWorkflows.ts
🚧 Files skipped from review as they are similar to previous changes (2)
- src/platform/telemetry/utils/templateBaselineStore.ts
- src/platform/telemetry/utils/classifyTemplateChange.ts
…-json fallback Per CodeRabbit feedback: previously the test added a setTemplateBaseline mock but did not assert it was called with the expected baseline, so the new baseline-capture behavior could regress silently. - assert active-state baseline path uses changeTracker.activeState - assert fallback path uses the fetched JSON when no active state exists - restructure workflowStore mock to use vi.hoisted so activeWorkflow can be swapped per-test
|
Addressed the CodeRabbit follow-up nitpick in d299033:
The previous commit added a Fixed:
Test count for this file went 10 → 12. Full suite still green:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR Created by the Glary-Bot Agent
Summary
Adds a
template_change_typeproperty to the PostHogexecution_startevent so analytics can distinguish runs that explore variations (seed/prompt only) from runs that materially change the workflow graph — without needing to join workflow payload cloud data with PostHog data via temporal joins.Requested in
#metrics-questionsthread to unblock template usage analytics.How it works
useTemplateWorkflows.loadWorkflowTemplate, the original workflow JSON is stored in an in-memory map keyed by template name (templateBaselineStore.ts, bounded at 32 entries via FIFO eviction).execution_start, if the active workflow is a known template and a baseline exists,classifyTemplateChangediffs baseline vs. current state from theChangeTracker.activeState. Livenode.widgets[i].nameis consulted to label per-widget diffs as seed/prompt/other.getExecutionContextwrites the result intotemplate_change_typeon theExecutionContextpayload, which flows through the existingtrackWorkflowExecution → trackEvent → posthog.capturepipeline. No provider, registry, or call-site changes needed (Mixpanel, GTM also receive the field automatically).Classification values
unchangedseed_only/(^|_)seed($|_)|noise_seed/ichangedprompt_only/(^|_)(prompt|text|positive|negative)($|_)/ichangedseed_and_promptstructuralThe field is omitted when
is_templateis false, when no baseline was captured (template loaded before this change, or session restored after reload), or if classification throws.Files
src/platform/telemetry/types.ts— newTemplateChangeTypetype +template_change_typefield onExecutionContextsrc/platform/telemetry/utils/templateBaselineStore.ts— module-level baseline map with deep-clone storage + FIFO capsrc/platform/telemetry/utils/classifyTemplateChange.ts— pure diff functionsrc/platform/telemetry/utils/getExecutionContext.ts— wirestemplate_change_typeinto the existing template branchsrc/platform/workflow/templates/composables/useTemplateWorkflows.ts— captures baseline immediately after fetching template JSONclassifyTemplateChange.test.ts,templateBaselineStore.test.ts, and 4 new cases ingetExecutionContext.test.tsVerification
pnpm typecheck✅pnpm lint✅ (0 errors on changed files; pre-existing warnings inuseLoad3d.test.tsanduseWorkspaceBilling.test.tsare unrelated)pnpm format✅pnpm vitest run src/platform/telemetry/ src/platform/workflow/templates/✅ 202 tests passOpening as draft per request; self-review via
/reviewto follow.Limitations / follow-ups
seed,noise_seed,text,prompt,positive,negative,positive_prompt,negative_prompt) and conservatively classifies unknown names asstructuralrather than misattributing.reduceAllNodes(same as node counts) so seed/prompt detection inside subgraphs works the same way.┆Issue is synchronized with this Notion page by Unito