How can a queue set itself to run again?

[mattjis]

In Vercel we have a 60 second limit on execution. I want to set up a timer that checks how we're going for time and whether we need Quirrel to call us back again.

I've tried the following, but am not having any luck.

import { Queue } from "quirrel/next"

export default Queue(
	"api/example",
	async (id) => {
		
		// processing
		// ..
		
		// oh no, we're running out of time. let's start again with a fresh 60 seconds
		await this.enqueue(
			id,
			{
				id: id,
				exclusive: true,
				override: true,
				runAt: new Date()
			}
		)
		
	}
)

How can this be done?

[Skn0tt]

You can access the queue itself by the following:

const queue = Queue(
	"api/example",
	async (id) => {
                await queue.enqueue(...)
	}
)

export default queue

You'll want to think about the following:

1. From your snippet, I get the impression that you want to update the currently running job (id and override: true). This likely won't work as you expect it to. Create a totally new job instead by using a random id (or leave it blank).
2. Enqueueing a job means making a call to the Quirrel API, which itself can take variable amounts of time. This could break easily! If possible, it's preferable to create all required jobs in advance instead of dynamically allocating new ones during execution.

[Skn0tt]

I assume you have some network-intensive processing step that takes a while to call a 3rd party API. (If it were CPU-Intensive, you'd know about the attempts needed, right?)

If either Quirrel's API or the 3rd party API has degraded performance (or even worse: Vercel's outgoing network is degraded), it would skew the timeouts you've chosen, and would lead to 60secs elapsing before Quirrel takes in the new job.

This would fail the original execution though, so it should be fine as long as you specify a retry policy.

[mattjis]

Okay I think I understand what you meant by that. The time difference between each of the calls isn't a huge issue.

Would I be better off using delay: 1 instead of runAt: new Date()?

[Skn0tt]

I sent my answer before your detailed explanation arrived for me. With that new information: Yes, your way seems fine :) When somebody asks about "reenqueueing if something goes wrong", I always think about flaky 3rd party APIs (somebody once told me about some API that fails 30% of the time and takes like 40secs to reply). But working with a payment provider, archaic response times aren't to be expected, I guess.

[Skn0tt]

About delay: 1: Just leave the runAt field free, it'll execute immediately then. (Conceptually the same as runAt: new Date() really, but that'd leave some room for clock drift and timezone issues)

[mattjis]

Perfect, thanks for the help!