Flows - passing returned data to the next job

[sam-hajj]

Example: To build a car assembly line, I will use a BullMQ flow, which would suppose the following code:

const flowProducer = new FlowProducer();
const queueName = 'assembly-line';
const chain = await flowProducer.add({
      name: 'car',
      data: {},
      queueName,
      children: [
             {
                    name: 'chassis', 
                    data: { 
                         type: '4x4' 
                    },
                    queueName 
              },
              {
                    name: 'wheels', 
                    data: { 
                         tires: 'snow tires',
                         package: 'premium'
                    },
                    queueName 
               }
      ],
});

Suppose that the data from the chassis child-step returns relevant metadata to the next step. What is the ideal way to pass this data easily to the wheels child-step? I would hope it would be something like the following:

const flowProducer = new FlowProducer();
const queueName = 'assembly-line';
const chain = await flowProducer.add({
      name: 'car',
      data: {},
      queueName,
      children: [
             {
                    name: 'chassis', 
                    data: { 
                         type: '4x4' 
                    },
                    queueName 
              },
              {
                    name: 'wheels', 
                    data: { 
                         tires: 'snow tires',
                         package: 'premium'
                         lastStepData: await flowProducer.Steps.Previous.returnvalue // returned value is `relevant metadata to the next step`
                    },
                    queueName 
               }
      ],
});

What would also be interesting is the ability to get the value of for example the first child-step.

const flowProducer = new FlowProducer();
const queueName = 'assembly-line';
const chain = await flowProducer.add({
      name: 'car',
      data: {},
      queueName,
      children: [
             {
                    name: 'chassis', 
                    data: { 
                         type: '4x4' 
                    },
                    queueName 
              },
              {
                    name: 'wheels', 
                    data: { 
                         tires: 'snow tires',
                         package: 'premium'
                         previousStepData: await flowProducer.Steps.Previous.returnvalue // returned value is `relevant metadata to the next step`
                    },
                    queueName 
               }
               // Many other steps
               {
                     name: 'wheels', 
                     data: { 
                          tires: 'snow tires',
                          package: 'premium'
                          firstStepData: await flowProducer.Steps.FirstChild.returnvalue // returned value is `relevant metadata to the current step`
                     },
                     queueName 
                }
      ],
});

There can be other methods similar to flowProducer.Steps.FirstChild.returnvalue, that could resemble CSS's pseudo-classes in some sense, like first-child, nth-child(), as long as they are legal within the library framework.

[sam-hajj]

On further inspection of the source code and documentation and trying to understand the philosophy of the development, I think my question/proposal doesn't even fit into the vision of the Flows feature, though it may be possible to hack together some solution. While reading, I kept seeing the heavy emphasis on parent-child hierarchy:

1. This class allows to add jobs with dependencies between them in such a way that it is possible to build complex flows. Note: A flow is a tree-like structure of jobs that depend on each other. Whenever the children of a given parent are completed, the parent will be processed, being able to access the children's result data. All Jobs can be in different queues, either children or parents, Documentation

2. BullMQ supports parent - child relationships between jobs. The basic idea is that a parent job will not be moved to the wait status, i.e. can be picked up by a worker, until all its children jobs have been processed successfully. Apart from that, a parent or a child job are no different from regular jobs.
   This functionality enables the creation of flows where jobs are the node of trees of arbitrary depth. Documentation

This leads me to believe the devs probably designed this to be used in such a way that there would be as many parent-child relations as necessary to unlock the data along the way instead of passing data from one child-step to the other. At first this was difficult to understand, but I'm guessing the idea is to have the child jobs act as collectors of data, which then the parent job can process if that makes sense

[dturton]

Did you find a solution to this?

[manast]

@dturton what is it that you need to do?