Azure Service Bus Support for MessageBrokers packages

[mumby0168]

I really like the way that the message brokers packages are broken up & how they work with RabbitMQ. I would be super keen to look at adding azure service bus as another implementation following the same sort of topology.

I'd love to help tackle this with your guy's support. I maintain a wrapper aiming for a similar thing at my work that we have been using in production for a while.

Happy to discuss if you guys had any thoughts on this already.

[spetz]

It's been one of the features that we've been asked about a lot of times, so yes - that's a great idea to add such kind of integration into Convey :)

[mumby0168]

It's been one of the features that we've been asked about a lot of times, so yes - that's a great idea to add such kind of integration into Convey :)

Cool, is there any design around topology you did for rabbitMQ?

Would you be happy to review some PRs for supporting this? Would it be you'd want to discuss before hand?

[GooRiOn]

Hi,
we could probably go for topic topology with subscription filter based on message property. PR is more than welcome :D

[mumby0168]

Hi, we could probably go for topic topology with subscription filter based on message property. PR is more than welcome :D

Okay cool, there is a bit of a performance hit using sub filters tho right?

[GooRiOn]

AFAIK yes, but the question is how would we map 1:1 approach from Rabbit in which queue is for each event type (routing key = event_type_name)?

[mumby0168]

AFAIK yes, but the question is how would we map 1:1 approach from Rabbit in which queue is for each event type (routing key = event_type_name)?

So if we are talking SQL Filters, they are the ones with the hit on Perf, we could use the correlation filters which offer up a To property so we could use that as our routing key almost?

https://docs.microsoft.com/en-us/azure/service-bus-messaging/topic-filters#filters

What do you think? I still need to review how the current implementation set's up RabbitMQ but I think this may do the trick.

[mumby0168]

Do you have any designs or rough sketches of the topology you have gone for with RabbitMQ, not probs if not I am sure the code will tell me 😃

[GooRiOn]

Correlation Filters is the one I was thinking of :D In terms of design diagrams... nope, BUT!

We usually went for the (configured by default) approach, in which you have:

• exchange per service (topic type)
• queue per event name
• therefore, routing key was just event type name (sanke_case)

But it'd be good to dive into the code (especially in subscriber) since there are few things that should be discussed:

• naming conventions for queues/exchange/routing keys
• dynamic queue/exchange creation
• pipelines on message received (i.e. for Jaeger plugin)
• DLX support
• retries policy etc.

There's probably more, but @spetz could provide more info I guess :D

[mumby0168]

Okay, that makes sense so an equivalent in ASB might be (at a high level):

• single topic per message
• subscriber per service
• then use the To field if it was a command aiming for a specific service (or we can use the auto-forward to a queue for these ... maybe)

Does that sound about right?

[spetz]

Yes, you can set pretty much any kind of RabbitMQ binding, either by using [MessageAttribute] or by providing your own implementation of IConventionsBuilder.

[mumby0168]

Yes, you can set pretty much any kind of RabbitMQ binding, either by using [MessageAttribute] or by providing your own implementation of IConventionsBuilder.

Okay, there would have to be another IConventionsBuilder for the SB package tho right? one to define topics/queues/subs

[mumby0168]

Okay @spetz @GooRiOn , so I have been thinking about this. I think this is my proposal:

Events

We have a topic per message type. This topic then has a set of subscribers which our case which be the name of the service defined for example we might have an OrderDetailsUpdatedEvent raised by the orders_service. This event let's say is subscribed to by the customer's service.

This would result in ASB as the following:

• topic name = order_details_updated_event
• one subscriber called customers_service another subscriber might be the accounts_service

This I feel follows good practice, we have a topic per message type, which I like. It also means giving an event you can see all the subscribers to it by the list of service names. We could also via config enable the subscribers to forward to queues taking maybe the convention of orders_service_order_updated_event_queue or something similar (this bit is maybe for the future).

Commands

For commands, I think we could follow a similar pattern. Let's say a CreateOrderCommand command for example. This would be processed by the orders_service again. This could take the form of something like below.

• topic name = create_order_command
• one subscriber = order_service with a correlation filter where To system property would have to equal orders_service.

I feel this gives us a good way to use commands via topics and a way to specifically state you are sending a command by setting the To field. I think it's good to use a topic in this case as we could have another listener which maybe audits messages into a db.

Summary

I think the above ^ should be the default. There is also the potential to have events and topic names to be prefixed with the service name that publishes to the topic for the example above something like orders_service/order_details_updated_event again we could control this via config. Again we can use the conventions builder to override this as and when.

Let me know what you think was this your intention? I really liked the fact we stick to a message per topic here. If you do like it I am happy to add the extra library. Would we want to do small PRs into main and then not publish the package yet or would we want a feature branch says features/message_brokers_azure_service_bus then I can branch from that and you guys review the PRs into that?

Let me know :)