Make workflow alerts conditional with severity-graded rules

[Paul Lizer (paullizer)]

Problem

A workflow carried a single alert_priority field (none | low | medium | high). In functions_workflow_runner.py, _create_workflow_priority_alert() was called unconditionally on both the success and failure return paths, so if the priority was anything other than none a notification fired on every run.

The result: the alert only ever meant "the workflow ran," not "something worth your attention happened." Owners either drowned in notifications or turned alerts off entirely.

Approach

Replace the single switch with a rule engine. A workflow now declares why it should notify and how loudly, and a run that matches nothing produces no notification at all.

run finishes (completed / failed / cancelled)
        │
        ▼
build_workflow_alert_facts()  ──►  run status, task results, final output, task outputs,
                                   File Sync result, agent signals, error text
        │
        ▼
evaluate_workflow_alert_rules()
        │   deterministic rules first (free)
        │   model-judged rules batched into one call, skipped when already outranked
        ▼
decision { should_alert, severity, category, delivery, matched_rules[], reasons[] }
        │
        ├── should_alert = False  ──►  no notification, decision recorded on the run
        └── should_alert = True   ──►  create_workflow_priority_notification()

Conditions

run_status, task_status, text_match (contains any / contains all / does not contain / regex), file_sync outcome, no_output, agent_signal, and model_evaluation — a plain-English condition such as "any certificate expires within 14 days" judged by a model.

Each rule scopes to the final output, any task output, or one specific task.

Severity, category and delivery

Severity grew from low/medium/high to info / low / medium / high / critical. Info and low land quietly in the notification bell; medium and above open the pop-up, overridable per rule.

Separately from severity, every alert carries a category. Run errors, task failures, File Sync failures and empty runs are categorized failure, which swaps the icon, forces the danger accent, and changes the badge from HIGH PRIORITY to HIGH FAILURE. This keeps "the workflow broke" visually distinct from "the workflow found something," at whatever severity the owner chose.

When several rules match, the highest severity wins and the alert opens with a Triggered by section listing every matched rule and its reason. One notification per run, always.

Agent-raised alerts

Agent-runner workflows can raise a signal mid-run through a new gated raise_workflow_alert plugin function, backed by a run-scoped contextvar that the runner opens around the run and resets in a finally block. An agent_signal rule then decides whether it notifies anyone; the rule's severity is a floor the agent can escalate above but never quiet below.

Notable design decisions

• New functions_workflow_alerts.py imports neither the runner nor the workflow CRUD modules, and takes the model client as an injected callable. Both save paths, the runner and the activity view share it with no import cycle, and it stays unit-testable without Azure credentials.
• raise_workflow_alert is opt-in (default_enabled: False). Registering it default-on would have silently granted every existing agent the ability to create notifications. Verified existing enabled-function lists are byte-identical.
• Model-judged conditions are cost-bounded: all model rules batch into a single call per run, and the call is skipped entirely — no model client is even resolved — when a deterministic rule already matched at or above their severity. Deterministic-only workflows add zero model calls.
• ReDoS guard on regex conditions: 200-character cap, compiled at save time so bad patterns fail on save rather than mid-run, nested-quantifier shapes like (a+)+ rejected, and searched text truncated.
• Each run records its alert decision so owners can answer "why didn't it alert?", surfaced in the workflow activity view. Costs one extra write per run, which felt proportionate for a rules engine whose main failure mode is silent confusion.

Migration

No data migration runs. Workflows carrying only alert_priority are migrated on read into two editable rules — Run failed → high and Run completed → <previous priority> — that reproduce the previous behavior exactly, including always opening the pop-up and staying silent on cancelled runs. Owners can then prune the noisy rule. Clients that predate alert rules, such as the create_personal_workflow plugin function, get the same treatment on save.

The legacy every_run mode also remains selectable in the UI.

Validation

• 26 new functional tests pass across three suites covering every condition type, scoping, severity resolution, failure category, validation, legacy migration, model-evaluation batching and the skip optimization, on_error handling, signal gating and the opt-in default.
• Zero regressions. Stashed the changes and ran the full workflow + notification functional suite as a baseline, then re-ran with the changes applied — identical pass/fail set. The ~20 failures are pre-existing and all stem from missing Cosmos credentials in the dev environment, not from this change.
• Route policy tests pass; no new routes were added, since the save normalizers are the only gate.
• All modified Python and JavaScript files parse.
• Updated one assertion in test_workflow_priority_alerts.py for the changed _build_workflow_alert_content signature, and the workflows-tab UI test for the new alert mode control.

Follow-ups deliberately left out of scope

• Per-rule cooldown / dedupe. A condition that persists across a short schedule will alert every cycle. This is the most likely next request.
• Digest-style rollups.
• A global admin kill switch for model-evaluated conditions, if a tenant-wide cost valve is ever wanted.

Docs

• New feature doc: docs/explanation/features/WORKFLOW_ALERT_RULES.md
• Release notes updated under v0.250.209
• VERSION bumped 0.250.208 → 0.250.209