What is the best approach to create a fork join flow?

[fabio-filho]

I've built a workflow that creates a sequence of jobs.
A job needs the previous one to be finished to be processed.

There is a model called fork-join that would solve my problem.
https://en.wikipedia.org/wiki/Fork%E2%80%93join_model

I implemented the workflow using FlowJob to easily connect a job with its dependencies.
https://docs.bullmq.io/guide/flows

My main problem is adding new child jobs dynamically on a pre-existing FlowJob and handling the errors dynamically too without breaking the base workflow.

An example

interface FlowJob {
  name: string;
  queueName: string;
  data?: any;
  prefix?: string;
  opts?: Omit<JobsOptions, 'parent' | 'repeat'>;
  children?: FlowJob[];
}

const childSubForkOne: FlowJob = {
   [...]
   name: "childSubForkOne",
   failParentOnFailure: true
}

const childForkOne: FlowJob = {
   [...]
   name: "childForkOne",
   children: [ childSubForkOne ],
   failParentOnFailure: true
}

const childForkTwo: FlowJob = {
   [...]
   name: "childForkTwo",
   failParentOnFailure: true
}

const childFork: FlowJob = {
   [...]
   name: "childFork",
  children: [ childForkOne, childForkTwo ]
}

const childOne: FlowJob = {
   [...]
   name: "childOne"
}

const childTwo: FlowJob = {
   [...]
   name: "childTwo"
}

const baseJob: FlowJob = {
   [...]
   name: "base",
  children: [ childOne, childFork, childTwo ]
}

1. As I said above, this won't work because I cannot add jobs dynamically. Each "fork" step above only knows the number of child forks after the previous job is completed;

2. In the current implemented code, each fork step, like above, needs to check every 2 seconds if all its children's jobs were finished;

3. I need the failParentOnFailure on fork children jobs to skip the jobs that need the previous ones to be completed successfully;

4. I tried to implement this workflow using the Process Step Job pattern describe below. The Waiting Children and the Delayed job don't work as expected when the child job got an error and, because of this, the workflow got stuck.
   https://docs.bullmq.io/patterns/process-step-jobs

[fabio-filho]

@manast, do you know how to solve this?

[manast]

Can you provide a concrete runable example?