Support deferred messages or delayed processing

[fgheysels]

We might have processes where we receive a message from servicebus that cannot be processed yet. (For instance because we're waiting on another message coming from another party first).
In such situations, it would be good if we could defer the message or resubmit the same message.
It would be nice if we could do this in the implementation of the message-handler. I can see that the AzureServiceBusMessageHandler now already has functionality to Complete or Abandon messages via protected methdos. Some 'defer' or 'resubmit' logic there would be appropriate I think.

I think there are 2 approaches:

Re-submitting:

I'm thinking of having the possibility of re-submitting the exact same message to Service Bus again, but with a specific / specified
ScheduledEnqueueTimeUtc property. Docs on this here. (That would probably mean completing the initial message and then resubmitting a copy of it , with all the same properties etc... but with a specified ScheduledEnqueueTimeUtc ?
(Or can this be implemented by abandoning the message and passing in the necessary 'propertiesToModify' ? [ pass in ScheduledEnqueueTimeUtc as propertyToModify ? )

Deferring

This might be another possible approach, but I think it's more complex?
Some documentation on the defer-functionality that is provided by Azure ServiceBus can be found here.
The drawback of deferring -as I see it-, is that you need another mechanism to retrieve deferred messages again, and that you need to specify which deferred message you want to retrieve again (so you need to keep state / administration for this).

[stijnmoreels]

That would be a cool feature, indeed. I see some similarity with #25 and #36 .

[fgheysels]

That would be a cool feature, indeed. I see some similarity with #25 and #36 .

I see some similarity with #25 indeed. However, @tomkerkhove mentions there that the pump should orchestrate this, but I don't know how the pump could do that. The pump has no knowledge of the 'business logic' that must be executed (what must be executed first).
Therefore, my assumption would be that the message-handler would be responsible for 'resubmitting' the message if it cannot be processed yet.

[fgheysels]

I've tried implementing this by calling the AbandonAsync method and specify a scheduledEnqueueTimeUtc property while abandoning, but this doesn't give us the desired results.

This issue is related to what we want to achieve here: Azure/azure-service-bus#454

One of the workarounds that is proposed in the above issue is to 'Defer' the message and keep track of the deferred message by sending another message to the queue which contains the sequence-number of the deferred message so that it can be retrieved later.
I think this might be a good approach for a work around.

I was thinking of having something like:

• a (protected) method in AzureServiceBusMessageHandler that you can call to 'Defer' or re-submit a message using a method that has a signature like this: DeferMessageAsync( DateTimeOffset resubmitDateTime);.
  When this method is called it should:

  • Defer the incoming message using the ProcessMessageEventArgs.DeferMessageAsync method.
  • Post a new message on ServiceBus with the necesary meta-data (amongst others the SequenceNumber of the received message that was deferred. The scheduledEnqueueTimeUtc property of this new message must be set to the resubmitDateTime value that was passed as a parameter.

• When the pump / messagerouter receives the message that has been posted because of the deferral, the messagerouter should be able to retrieve the original deffered message via the ReceiveDeferredMessageAsync method and the sequenceNumber that was stored in that message. Once the deferred message is retrieved, it should be routed to the correct MessageHandler, just as we're doing now with 'normal' messages.
  (I believe for this to work, we would need to be able to register a message-handler that can handle those messages that have been posted because of a deferral ? Maybe this can be done by wrapping the necessary information in a DeferredMessageMetadata type or something ? When such a type is received, the messagehandler that is linked to this type can be executed. The implementation of this message-handler retrieves the deferred message, and makes sure that the messaage is routed ? Just thinking out loud).

Of course, this is still -imho- a workaround for the problem at hand. Ideally, MSFT solves this by adding this functionality in ServiceBus as this work-around brings a few other problems to the table:

• we must be able to send messages from a message-handler to ServiceBus. This means that the process that is processing messages would also need to have 'send' rights to Service Bus.
• What if the Message is successfully deferred, but sending the scheduled message fails ?
• What about topics and subscriptions ? See this side-effect of this workaround (and this possible resolution to it).

[stijnmoreels]

Sounds like a plan! 🎉 Thx a lot for documenting/investigating this so thoroughly. Will see if I can work on something while you're gone. 👍

[stijnmoreels]

The feature is done being designed and almost picked up for development: Azure/azure-service-bus#454 (comment)
Maybe we can wait a bit for it.

[fgheysels]

Well, I think this is something we're going to need very soon in a project. Also, other teams might find it very useful.
I don't know how soon MSFT will start development, and how soon it will be released / available. I guess we can close that gap for now using our 'work around implementation', and then, once MSFT has provided a cleaner solution, modify our implementation to make use of the feature that MSFT has developed ?

[stijnmoreels]

Yes, think that is a good approach. Unless of course this is very critical for other projects, otherwise I would be in favor to wait a bit for Microsoft to catch up.

[fgheysels]

I think it is a crucial feature, something that should have been implemented by MSFT from the start imho.
I'm afraid it can take up to 6 months or more before MSFT will deploy this.

Another thing regarding the possible implementation. I've discussed this with @gverstraete yesterday as well, and his first idea on how to implement this, could also be an option:

When 'deferring' a message, we could create a clone of the message, (with the same body and properties) and send this message to service bus with a `ScheduledEnqueuetimeUtc', and next to that we must then also make sure to 'accept' the original message so that it is no longer available in the queue.

[stijnmoreels]

Ah yes, but I'm wondering what Impact this could have on the performance/scaling/cost of Service Bus as for every message, there will be two in the queue.

Otherwise, this is indeed a rather elegant solution.
If we're thinking about implementing this, we should also probably note somewhere that the authorization of the Service Bus resource will require Write (new messages) too, instead of Read only.

[fgheysels]

Ah yes, but I'm wondering what Impact this could have on the performance/scaling/cost of Service Bus as for every message, there will be two in the queue.

Service Bus bills you per operation, not by the amount of messages. Service Bus Pricing.
Of course, in a situation where you need to retrieve a deferred message, that 's an additional operation, but I would say that additional costs for this would be marginal ?

Otherwise, this is indeed a rather elegant solution. If we're thinking about implementing this, we should also probably note somewhere that the authorization of the Service Bus resource will require Write (new messages) too, instead of Read only.

That's a good point, and indeed a drawback.

[fgheysels]

The thing is that we'll need to decide on how to continue with this:

• implement using deferring the message and enqueue a message that says 'retrieve the deferred message'
• implement via the approach that 'clones' the original message and sends it with a scheduled date, abandon the original message
• wait for MSFT

I would be in favor to not to wait for MSFT as we don't know how soon the feature will be delivered. We can close the gap for now with an alternative approach. Once MSFT comes up with their solution, we can rework our workaround to use the 'official implementation'.

What's your take @gverstraete ?
This is a feature we'll certainly require at a project I'm working on with @jcools85 right now, so he might have some ideas as well ?

[gverstraete]

The thing is that we'll need to decide on how to continue with this:

• implement using deferring the message and enqueue a message that says 'retrieve the deferred message'
• implement via the approach that 'clones' the original message and sends it with a scheduled date, abandon the original message
• wait for MSFT

I would be in favor to not to wait for MSFT as we don't know how soon the feature will be delivered. We can close the gap for now with an alternative approach. Once MSFT comes up with their solution, we can rework our workaround to use the 'official implementation'.

What's your take @gverstraete ? This is a feature we'll certainly require at a project I'm working on with @jcools85 right now, so he might have some ideas as well ?

Agree, I think cloning makes most sense

[fgheysels]

For implementation, I would go with 'cloning' the message. That looks to be the most simple solution.

Next to that, we must take into consideration that a processor not only needs to have listen permissions to the queue (or topic), but also needs to have send permission.
In my opinion, that is something that we can add in the documentation. When a message is being deferred (send back to the queue), and the connection that we're using doesn't have send rights, we'll just throw an exception.

When MSFT has finished it's implementation, we can refactor our internal workings to make use of the functionality MSFT provides. If no 'send' rights are required for MSFT's implementation, we can modify our documentation and make a note of this in our release-notes.

[jcools85]

Any update on this? We need this feature at NxtPort to implement a retry mechanism with custom backoff times.

[stijnmoreels]

Any update on this? We need this feature at NxtPort to implement a retry mechanism with custom backoff times.

We are working on a circuit breaker, but that would not include deferred messaging.
See also this feature request on Microsoft's Service Bus, as there is some missing functionlity.

No time/budget to implement a custom solution as of yet. But we are always happy to receive PR's 😉.

[stijnmoreels]

I think it is possible, though, to defer the message yourself in the message handler, add an application property with a timestamp, and filter with another message handler for certain past times. But as I've said, no room to take anything else at the moment.

[fgheysels]

I think it is very hard to implement this in the MessageHandler itself. First of all, Arcus currently doesn't expose a DeferMessage method or something similar. Next to that, the 'application developer' would need to be able to retrieve the deferred message (that wouldn't be that difficult), but the developer would need a way to send this message through the Arcus message-pump.

It looks like MSFT is not very responsible on my requests in this issue so my plan for a workaround in Arcus would be like this:

1. Introduce a DeferMessageAsync(TimeSpan delay) method. This method would call the DeferMessageAsync method of the `ServiceBusReceiver
2. Since we're now responsible for retrieving the deferred message ourselves; the DeferMessageAsync method of Arcus should send a scheduled message to servicebus which contains the details of the deferred message. (It definitely needs the SequenceNumber of the deferred message).
3. An Arcus ServiceBus messagepump must have a default MessageHandler (or another mechanism) for receiving and processing the scheduled message that we've sent in step 2. This mechanism must make sure that it can interpret the
   message, retrieve the stored SequenceNumber and retrieve the deferred message via the SequenceNumber using the ReceiveDeferredMessage method of the ServiceBusReceiver.
4. The Arcus MessagePump would then also need to make sure that the received deferred message is sent to the Arcus MessageRouter so that the correct registered message-handler can process the message.

More info on deferring in ServiceBus can be found here: https://learn.microsoft.com/en-us/azure/service-bus-messaging/message-deferral#message-deferral-apis. The approach I mentionned here is actually more or less described in the 'sample scenarios' paragraph of that article.

[stijnmoreels]

We're talking about the same thing here 😉

[fgheysels]

We cannot do that in the messagehandler.
We currently have no access to the ServiceBusMessage itself, nor do we have a way to make sure that the deferred message that we would then retrieve, is send through the message-router.