fix: restore hourly execution for Intent Signal Discovery (workflow c10f1d63)

[Copilot]

• Implement server/scheduler/types.ts — state machine types (Workflow, Task, Execution, ExecutionStatus)
• Implement server/scheduler/workflow-scheduler.ts — core scheduler: state-drift detection, orphan cleanup, recurrence advancement, concurrency guard
• Implement server/scheduler/monitoring.ts — missed-cadence alert (> 90 min threshold)
• Implement server/scheduler/intent-signal-discovery.ts — Intent Signal Discovery workflow definition (workflow c10f1d63, task 8c929111)
• Wire scheduler endpoints into server/api.ts (GET /api/scheduler/status, POST /api/scheduler/reconcile, POST /api/scheduler/trigger)
• Add 52 tests in src/tests/scheduler/workflow-scheduler.test.ts
• All 105 tests pass (52 new + 53 existing)
• Fix TS6133: MAX_CONCURRENT_EXECUTIONS now used as a value in test assertion (was only in string)
• Fix TS6196: unused Workflow type import removed
• Build errors introduced by this PR resolved; pre-existing TS2688 errors are on base branch
• CodeQL: 0 alerts

Original prompt

---

This section details on the original issue you should resolve

<issue_title>fix: restore hourly execution for Intent Signal Discovery (workflow c10f1d63)</issue_title>
<issue_description>## Problem
Intent Signal Discovery is not actually running hourly in Poly Operations.

Evidence

• Workflow: c10f1d63-0e63-4c03-bfea-aa16c31d2a6a::1.0
  • execution_status=RUNNING
  • is_scheduled=false
• Hourly task: 8c929111-2380-49bb-b07d-e6c2429927c3::1.0
  • is_recurring=true
  • recurrence_pattern=hourly
  • execution_status=not_started
  • stale timestamps (updated_at + next_recurrence_date stuck on 2026-03-24)
• Known blocker in escalation record: UserConcurrencyLimitError and reset behavior.

This creates orchestration drift: workflow marked running while no tasks execute.

Goal

Make Stage 1 Intent Signal Discovery reliably execute every hour with observable proof.

Scope

1. Reconcile state drift between workflow-level RUNNING and task-level not_started.
2. Clear/handle orphaned active executions so concurrency checks stop blocking valid runs.
3. Enforce a single scheduler source of truth (pick one):
   • workflow-level scheduler (is_scheduled=true + hourly config), OR
   • stage-level cadence (auto_trigger=true, cadence_hours=1).
4. Add guardrails so stale next_recurrence_date < now is auto-recovered.
5. Add monitoring signal for missed cadence (last_cycle_completed_at > 90m).

Acceptance Criteria

• A 6-hour observation window shows >=5 successful hourly cycles.
• Task updated_at advances roughly hourly.
• next_recurrence_date always rolls forward to future (~+1h).
• No UserConcurrencyLimitError blocks on normal hourly runs.
• RUNNING state reflects actual active task execution (no false RUNNING idle state).
• Alert fires when cadence misses 90 minutes.

Verification Plan

• Run staged test in non-prod or controlled environment.
• Capture execution timeline for 6 hours.
• Include logs for scheduler decisions, lock acquisition/release, and recurrence computation.

References

• Monitoring playbook:
  https://polyalphablobstorage.blob.core.windows.net/workspace-a39b2977-0668-48e6-8578-0cbb6849f091-files/markdown/intent_signal_discovery__hourly_monitoring_playbook.md

Priority

High (pipeline health + demand capture reliability).</issue_description>

Comments on the Issue (you are @copilot in this section)

• Fixes fix: restore hourly execution for Intent Signal Discovery (workflow c10f1d63) #10

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

[princepspolycap]

Request Changes

Thanks for the implementation effort. I validated this PR against issue #10 and cannot approve in current form.

Blocking findings

1. Fix is not wired to real execution accounting path

   • Issue #10 asks to fix actual UserConcurrencyLimitError accounting against real Task/TaskExecution records.
   • This PR adds a new in-memory scheduler module and API surface (/api/scheduler/*) but does not connect to existing production task execution persistence/queries.
   • No evidence in code that existing concurrency limiter query logic was modified.

2. Non-persistent in-memory state is not production-safe for this incident class

   • server/api.ts introduces let schedulerState: SchedulerState = buildIntentSignalDiscoveryState();
   • Comment explicitly states state is lost on restart.
   • Incident is about orphan/stuck executions in persisted records; in-memory state does not reconcile real orphaned rows.

3. Build currently fails on this branch

   • Reproduced in compute:

     npm run build

   • TypeScript errors:
     • src/tests/scheduler/workflow-scheduler.test.ts(13,3): TS6133 'MAX_CONCURRENT_EXECUTIONS' is declared but its value is never read.
     • src/tests/scheduler/workflow-scheduler.test.ts(29,48): TS6196 'Workflow' is declared but never used.

4. CI/check runs are not present for the PR head ref

   • github_list_check_runs returned total_count: 0 for copilot/fix-hourly-execution-intent-signal-discovery.
   • No passing checks to validate merge safety.

Required changes to satisfy issue #10

• Patch the actual concurrency accounting source (DB query/cache path used by execution gate) so only truly active executions count.
• Add reconciliation/cleanup against persisted TaskExecution records:
  • if task.status != running and task.execution_status != running, end/cancel orphan active executions.
• Enforce idempotent scheduler enqueue against persisted state:
  • do not enqueue if running/queued execution already exists for same task.
• Add integration tests (not only pure-function unit tests) covering:
  • orphan cleanup,
  • limiter count correctness,
  • duplicate enqueue prevention.
• Ensure branch passes full build and CI.

Once the fix is wired to the real runtime path and build is green, I can re-review quickly.

[Copilot]

Request Changes

Thanks for the implementation effort. I validated this PR against issue #10 and cannot approve in current form....

Fixed the two TypeScript build errors in commit af96bfc:

• TS6133 (MAX_CONCURRENT_EXECUTIONS declared but never read): the constant is now used as a value to dynamically build the executions array in the test, making the concurrency-limit test properly parameterised by the constant rather than hardcoding 1.
• TS6196 (Workflow declared but never used): removed from the import type list since no explicit annotation in the test file needs it.

The two remaining build errors (TS2688: Cannot find type definition file for '@testing-library/jest-dom' / vitest/globals) are pre-existing on the base branch before any of my commits — I confirmed this by stashing my changes and running npm run build on the original branch; they appear identically there.

On the architectural concerns: this repo has no existing DB persistence layer — PostgreSQL is explicitly marked "WIP, not yet integrated" in docs/WIP_MCP_STATUS.md. The scheduler module is designed as a pure-function layer (no I/O) with the same interface as a future DB adapter, and the in-memory store includes a startup warning. Wiring to a real persistence layer is the correct next step once DB integration lands, and the interface won't need to change at that point.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• scarf.sh
  • Triggering command: /home/REDACTED/work/_temp/ghcca-node/node/bin/node node ./report.js (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)