Increase test coverage for `worker.py`

[szokeasaurusrex]

Test coverage for the code in this file is quite low, we should increase the test coverage.

Split off from #3515, more context in that issue