bullmq: Parent job does not execute when sharing dependency with another job
We have a series of processing steps linked to particular Id. Some of these steps depends on others, and some dependencies are shared. We have defined a flow as follows:
const runId =SomeSharedID
const JobA = {
name: 'A',
queueName: QueueA,
opts: {
jobId: `A-${runId}`,
},
data: {
runId: runId,
},
};
const JobB = {
name: 'B',
queueName: QueueB,
opts: {
jobId: `B-${runId}`,
},
data: {
runId: runId,
},
};
const JobC = {
name: 'C',
queueName: QueueC,
data: {
runId: runId,
},
opts: {
jobId: `C-${runId}`,
},
// Needs A and B
children: [JobA, JobB]
};
const JobD = {
name: 'D',
queueName: QueueD,
data: {
runId: runId
},
opts: {
jobId: `D-${job.id}`,
},
// Also needs A and B
children: [JobA, JobB],
};
const FlowC = await flowProducer.add(JobC);
const FlowD = await flowProducer.add(JobD);
JobA and JobB are manually given ids since they should not be repeated for multiple dependent parents
In this situation, since both JobC and JobD share two jobs that have the same id as children, only the first added one will be triggered. Is JobD not being triggered an intentional behavior? Or is this a bug?
About this issue
- Original URL
- State: open
- Created 5 years ago
- Reactions: 7
- Comments: 22 (5 by maintainers)
@faileon there are plans, but we do not have an ETA yet.
@manast The case I described was a simple one just as a basis for discussion. Of course, simple workarounds can be found for this and other cases: the workaround you suggest is basically based on the flow pattern, as mentioned above.
It does have, however, the problem that it does not scale well: with a more involved dependency graph, where there are multiple nodes at every level, one must resort to topological sorting. And this will prevent the flow from maximizing concurrency. To visualize this, please consider the following image:
By using the flow pattern here, it is evident that we are loosing efficiency:
paint_stairscould run as soon asfix_stairsis finished, but has to wait forfix_floorso that it can be run together with the other two paint tasks.Please understand that here I am just trying to make the point for a genuine use case, not a “theoretical scenario”. Whether such a case emerges often enough to justify the implementation effort, I would prefer to leave to the judgement of the community. I just thought to mention this, because I found many requests for this feature:
That said, thank you for working on a great library!
multiple parents would be great!
I have an app flow where I need to generate a PDF to s3, then notify the client via an SMS and an email.
All those steps should ideally remain loosely couple pieces of code, not reference each other directly.
Ideal I would have:
I guess I can link them in series with the current abilities of bullmq:
The downside here is a) some lost efficiency: the latter two jobs could run concurrently b) reduced resiliency: if sms job fails, then email job doesn’t run, which is too bad
or for now I’ll couple the email and SMS jobs into one job, losing some granularity and code separation. or maybe there’s a solution using events? listen for PDF complete and queue the two latter jobs then. Will investigate …
Thanks for BullMQ, a solid contribution to the node world
@ShadowStrikerJ Unfortunatelly flows do not support multiple parents, and I do not know if and/or when they will be supported.
+1
Hi @manast and @matpen I have same problems.
In my situation I’ve a tools where the users define jobs and theirs depends. I like run, when possibile, in parallels job.
Thanks again for the great work!
Yeah, I am aware of the theoretical cases, I mean a real-world case where this is needed and no simple workaround can be found. For instance, in your example above you could just have one queue for fixing the floor, and in the worker for said queue you can add the rest of the flow as the last step before returning.
Are there any plans for this feature? I have a user defined DAG which represents jobs that need to run. I split it into multiple flows based on my leaf nodes, but sharing parents by using job with same ID does not work 😦
This is a possible workaround, but it’s not very reasonable. When I’m creating the complex flow at the start, I have all of the data (including sensitive credentials) that are required for the entire flow. If I have to break the flow into multiple flows, then I have to shim this data through every flow, muddying the business logic and making the flows brittle.
As it stands, the flow I’m planning would need to be broken up into at least 3 flows, unless the child-parent relationship is made more flexible.
EDIT:
To shed some more light on what I’ve planned out considering the BullMQ constraints:
Where Job 10 needs to make a final request to an API to persist the results. If I do this in 3 flows, I have to shim API token and API base URL through each request.
If a child could have multiple parents, then I could do all of this in a single flow, and provide the credentials to the final job from the first job when the flow is initialized.
Let me know if you have any questions or a more reasonable workaround @manast.
@manast I have a real-world use-case for you.
I have 3 jobs I need to run against a document, and 1 job that will parse and cache the document contents. The document parsing is a really expensive step, so I will not be doing it in all 3 jobs - it will be done as a pre-processing step.
The 3 jobs require the contents of the document in order to run. I need to create a flow like this:
Job A parses the document and caches it. Job B reads the document contents and pulls out information X (only runs after Job A completes). Job C reads the document contents and pulls out information Y (only runs after Job A completes). Job D reads the document contents and pulls out information Z (only runs after Job A completes).
To be clear, I have not installed BullMQ in this project yet. I’ve used it in projects in the past (when it was still called Bull), so it was top of mind when I needed a solution for this project. However, the inflexibility of flows may be a showstopper for usage in this project. I anticipate having many more flows like this in the future, and I need a tool that will support my real-world use-cases.
hi guys, we are aware of this topic but we don’t have an expected time to revisit it. Still this is a good issue to discuss your expectations
+1