[BUG] `"Scheduling timed out"` does not show as an error

[vnghia]

Describe the issue
When using aio_run_many and the children task can not run because of scheduling timed out, the status of the parent is still success. This is very misleading and could lead to bigger problem because the error is not addressed correspondingly.

Environment

• SDK: Python v1.33.5
• Engine: Selfhosted v0.86.7

Expected behavior
When the children task can not because of scheduling timed out, the status of the parent should show error

Code to Reproduce, Logs, or Screenshots

# set child_wf concurrency to 1
# spawn 100 child tasks
# sleep like 10s
# schedule timeout to 10s
child_wf.aio_run_many([])

Additional context
Thank you very much for Hatchet :D

[sajdakabir]

Hey @mrkaye97! Would love to take a shot at this one . I dug into the code and traced the issue:

• Scheduler detects SCHEDULING_TIMED_OUT and sends a cancellation message with an empty eventMessage
• CancelTasks in the repo creates a V1TaskEventTypeCANCELLED event with no ErrorMessage attached
• Dispatcher (server_v1.go:534) already has a // FIXME: this should be more specific for schedule timeouts comment here — it ends up sending res.Error = "" (empty string)
• Python SDK filters empty error strings as falsy, so the workflow appears successful

A couple of ways to fix it:

1. Engine-side — propagate the cancellation reason through CancelTasks so the proto error carries something like "Scheduling timed out" (also fixes the FIXME and improves the dashboard UX)
2. SDK-side — make Python SDK raise on workflows that finished with no outputs (more defensive but symptom-focused)

Curious which approach you'd prefer before I start! Happy to take whichever direction makes sense.

[mrkaye97]

I think we might have an open fix here: #3876

Mind taking a look there and seeing if it's in line with what you were thinking?

[sajdakabir]

Yep, that PR matches what I had in mind!