EIP – Message Channel Outbox Design Pattern

Summary

The outbox pattern is a good application design pattern to use when a Sender has trouble sending messages to an Event or Message Broker. Messages can be persisted to an application database, and then retrieved at a later time by an Event Processor.

Outbox Pattern

A client updates a CRM application. The record state needs to be transferred to other supporting systems (ERP/SCM etc). A Message Channel is used to transfer the event state. The record update is persisted in the CRM database, however, the event message was not transferred to the Message Channel due to network, application failure. The Supporting systems (ERP/SCM) are not updated; these systems now have stale Customer data.

This is a common occurrence in some organisations where the Sender application doesn’t have the correct routines built to handle exceptions.

I once worked on a project where the Sender application was situated in Brazil and the Message Bus (RabbitMQ) was hosted in a Virtual Machine in Germany. The Sender application would always time out during peak internet hours (6-10 PM). During this period it was common for families to get together to watch streaming services, and go online for gaming or casual browsers; either way, it would stretch the local ISP.

The Outbox integration pattern solves this issue to some degree. The Outbox pattern works on the following principles:

1. A transaction committed to the CRM database is persisted. This persistence represents a real event, like a Customer update. If there are no issues with communicating with the CRM; then the CRM database is used as an intermediary data store.
2. A new table is created in the CRM database called CRMOutbox.
3. When a record update operation is about to take place; two operations are committed and persisted to the database storage 1) the record details and 2) the full details of the message (Headers, Body, Payload etc.).
4. An Event Processor is set up to pull the message out of the CRMOutbox table, and to publish the message on the Message Channel at set intervals.
5. When a live connection is established or the Event Processor can publish a message to the Message Channel, the record from the CRMOutbox is deleted. This provides some guarantees that the message will eventually be submitted to the channel.

Known issues

There are some known issues with this approach, primarily concerning message duplication, the introduction of additional components and the transaction now being committed to the main production database. Some technologies like Microsoft Azure provide mechanisms for dealing with these issues. Refer to the technology stack being used.

Azure Cosmos DB Implementation

Microsoft has provided a working implementation of the Outbox Pattern. Read more about it here.