Concurrency #4
Description
I am using a bluebird-queue via NodeJS to queue HTTP endpoints as tasks. Each task has a 3-level Promise dependancy that has to resolve before it is complete.
One Task
GET -> endpoint 1 // returns promise
GET -> other endpoints in async // returns promise
POST -> final endpoint // return promise
I put 20,000 of these tasks into the bluebird-queue with queue.add() and then subsequently call queue.start(). All errors are caught and the handlers resolve the Promise so the task can complete.
I have set concurrency to 50. My initial expectation is that the queue will process 50 at any given time but instead it waits for the first 50 to complete before starting on the next 50.
Unfortunately, some of these Requests can take up to 10 secs to complete - and if a single request takes longer to finish, the whole queue stalls until the Promise resolves.
I have tested with promise-queue and it differs from bluebird-queue where the former processes tasks in the queue at max of 50 tasks at any given time instead of per 50 at a time.
Is this intended behaviour?
Here are my config settings:
var Queue = require('bluebird-queue'),
queue = new Queue({
concurrency: 50,
delay: 10, // ms
interval: 1 // ms not quite sure what this means
});
For your reference, here is the link to the stackoverflow question.