How to add a new concrete implementation of the event bus

Context

Here is a list of the existing concrete implementations of the event bus:

• Kafka

• Redis Streams

This how-to is to help you add a new concrete implementation, for example using Pulsar or some other technology.

Producing

There should be a producer class that inherits from EventBusProducer in openedx-events.

The defined send method is meant to be called from within a signal receiver in the producing service.

Consuming

At a high level, the consumer should be a process that takes the signals and events from the broker and emits the signal with the event. There should be a consumer class that inherits from EventBusConsumer in openedx-events.

The consumer class then needs to implement consume_indefinitely loop, which will stay running and listen to events as they come in.

We have included an utility function called prepare_for_new_work_cycle in openedx-events which needs to be called before processing any signal. Currently, it reconnects the db connection if required as well as clears RequestCache and there may be later, more comprehensive changes. These steps mimic some setup/teardown that is normally performed by Django in its request/response based architecture.

Checkout consumer.py in event bus redis implementation.

Abstraction tickets

The known remaining work for a fully abstracted event bus is captured in the Abstraction tickets