Message Design In An Event Driven Architecture

A few good arguments for adopting an asynchronous messaging pattern in a microservice environment include high scalability, loose coupling and better use of resources. Instead of continuously polling the source system, the flow of communication is inverted. Messages are produced by the source system in order to inform consumers of noteworthy events. Those events are expressed in messages that are distributed by a message broker. The design of those messages has a direct impact on the implementation of the producing and consuming services.

Message Design

Events are supposed to inform consumers that something has happened. In frontend applications that could either apply to an application state change triggered by an action or to a context change, an event captured by the browser, like a page scroll, cursor movements and there like. Without loss of generality, in a classic CRUD based backend application it usually relates to a state change: A record was inserted, updated or deleted.

Operations on Entities

A simple representation of an event is the full state of a data item and a fixed set of operations that can be applied to this state. This is quite similar to the application of the REST pattern, where a POST (creation) and PUT (update) HTTP request include the full payload of the entity to be inserted or updated. In order to inform the consumer about the nature of the event, metadata could be added to the event.

For a todo that was just added to a list of todos, the message representing such an event could look like this:

{
    "metadata": {
        "action": "INSERT"
    },
    "data": {
        "id": 1,
        "text": "My first todo.",
        "completed": false
    }
}

Let's imagine the owner of this todo has completed the task and an UPDATE message is published. It is structured in the same way, only now the action property is assigned to UPDATE and the completed property is set to true. For a consumer it is not clear, what triggered the event and what in particular changed. It is only obvious, that something has changed, and it is up to the consumer to find out what it was.

In some scenarios this could still be sufficient though: If it is not necessary to react to events but rather store the latest state for the purpose of extending the service's dataset, it should be fine to include the full state in each message. Choosing this approach also reduces the dependency between each event sine every event self-contained as it includes all the information to represent a data item's current state.

Occurrences of Actions

Another way to tackle the message design is to think of events as occurrences of actions in a well-defined process. Every action leads to state change. Therefore, each message should describe the action that occurred and the properties of the state that changed as a consequence.

A completed todo item could be represented for example like this:

{
    "event": "TodoCompleted",
    "id": 1
}

The consumer is now able to easily understand what happened ("event": "TodoCompleted"). The reaction to that event is entirely up to the consumer. It could mean updating the todo's state by setting the complete property to false.

In the end a series of events leads to the current state of an entity. This is called "event sourcing". Although the messages are much more succinct and carry a stronger semantic meaning as they represent an event very specifically, there are extra costs on the implementation side. It's up to the consumer to interpret those events and to react to them appropriately. There could be a multitude of events that are emitted, which makes documentation essential.

Order

Ideally, events are self-contained in a sense that they do not have any other dependencies towards other events and can appear in any particular order. This reduces complexity both for the message queue, which might otherwise be required to order messages by a certain criterion, and the consumer, that otherwise needs to make additional queries and decision based on the order of events.

If the text of a task has changed multiple times (e.g. ToDoTextChanged) and the events are not ordered, the consumer has no way of determining the most recent text of the todo item.

This can be solved by adding a criterion by which the consumer is able to order the sequence or decide on an appropriate action. If the service has already seen at least one event and the following one has a lower sequence number, it can simply choose to ignore this event for example. But this also means the service needs to query the latest sequence number or store it in memory, make the comparison, which introduces additional overhead.

Other Event Types

Events are pretty flexible in what they actually express. The word "message" was chosen deliberately, since it carries the notion of informing the consumers about an event. As soon as the message leaves the sender, the sender's job is done, and it has no idea how the consumers of the message will react to it. This is exactly what manifests the loose coupling between sender and consumer and enables the scaling as the sender can carry on with its duties, while the message broker distributes the message amongst the consumers.

There are other patterns, like the Command Query Responsibility Segregation (CQRS), pattern, that describes asynchronous communication based on "query" and "command" events. In contrast to plain messages, the sender sends a message with the clear intent to invoke a certain action or process on the receiver's side. Furthermore, the result of the invocation is even sent back to the original sender. In this case asynchronicity is they key selling point, but loose coupling only to a certain extent - if at all.