Flexible recovery from abnormal termination

[hajipy]

Jobs' state will be State.FAILURE if either:

• Queue.shutdown was timed out
• Your program suddenly terminated

User cannot know job failure reason.

1. By above behavior
2. User processor threw Exception

it is problem.

---

Consider a better behavior than in this Issue.

Plan A

Queue.createQueue accepts parameter how to recover.

const queue = await EmbeddedQueue.Queue.createQueue(
    { inMemoryOnly: true },
    { recoveryStrategy: RecoveryStrategy.FAIL }
    // { recoveryStrategy: RecoveryStrategy.REINITIALIZE }
);

Plan B

Add Queue.recoveryUnsettledJobs(job) method for recover.

const queue = await EmbeddedQueue.Queue.createQueue({ inMemoryOnly: true });

queue.recoveryUnsettledJobs(async (job) => {
    switch (job.type) {
        case "typeA":
        case "typeB":
            await job.reinitialze();
            break;
        default:
            await job.fail();
            break;
    }    
});

// after queue.process() called, you cannot call queue.recoveryUnsettledJobs().
queue.process(
    "adder",
    async (job) => job.data.a + job.data.b,
    1
);