Enterprise Integration Design Patterns Catalogue

Summary

This page details a list of accepted Enterprise Integration Patterns (EIP). All integration design patterns exhibit some common characteristics.

Well-defined Integration Architecture

In distributed systems, a well-defined integration architecture has components designed around the Open Systems Interconnect (OSI) conceptual model and the respective set of Internet Protocol Suits/RFCs including proposed specs outlined by the Internet Engineering Task Force (IETF).

A high-fidelity Enterprise Integration Pattern will involve the following loosely-coupled subsystems:

Building Blocks of EIP

Messaging Systems

The messaging system is the platform responsible for sending or receiving messages. In an event-driven architecture, a message sender platform might be a LOB application. The endpoint receiver does not need to be an active and responsive system, it can also be a data store.

Message Construction

Message Systems are responsible for constructing the message that will ultimately be sent down a Message Channel. The message construction process involves creating the message Header and the Body (Payload). Before constructing a message the following layers of the OSI model must clearly be understood; Application layer, Transport layer, Internet Layer and Link Layer. Web traffic typically flows through TCP/IP, however, if you are building a listening service for a high availability system, you’ll want to implement UDP over TCP.

An application creating a message should clearly define the message format. It’s typical for web services to communicate in JSON/XML format, however proprietary applications communicate with well-defined EDI formats for their industry such as AMQP, AS2, EDIFACT, X12, RosettaNet working with a multitude of application layer protocols (see the extensive list here).

Message Endpoint

A message endpoint is an interface between an application and the messaging channel. A message endpoint sits in two distinct categories; the sender endpoint and the receiver endpoint. Think of message endpoints as sub-systems to a core application. In Microsoft Power Platform for instance; the message endpoint is the SOAP/REST endpoint.

Message Channels

Channels are the plumbing for message transfer. Although messages are transported via TCP/IP which operates at the OSI 4 layer, messaging channels operate at the OSI layer 7 (Application Layer). The most common type of channel is point-to-point, where one application communicates with another application directly.

Message Routers

Message routing defines guidelines for routing messages to its target endpoint receiver. Routing can be as simple as sending all messages to a single receiver, or it can be dynamically configured based on several criteria and using several methods. For instance, a single event message is sometimes broadcast to dozens of receiver endpoints in high-availability systems.

In Apache Camel, for instance, the following routing rules define the endpoint systems:

from("direct:a")
    .choice()
        .when(simple("${header.crm} == 'salesforce'"))
            .to("direct:b")
        .when(simple("${header.erp} == 'oracle'"))
            .to("direct:c")
        .otherwise()
            .to("direct:d");

Message Transformers

Message transformation defines rules for transforming messages for additional processing or to match the expected document format of the service to which the message will be sent (see Message Construction). A message may require transformation because the structure of the received message is not in the expected standard or because the message must be converted from a non-standard format to XML. For example; a flat-file CSV document may need transforming into a serialised JSON before the message is transported on the Message Channel.

Although Message Transformation typically occurs on the Message Receiver endpoint, Message Transformation can be handled at either senders or the recipient’s endpoint (see Content Enricher).

System Management

Integrated systems need some capability to monitor, log, debug and manage workloads. Apache Fink and Airflow have a robust set of internal tools for system management. Microsoft provides several management tools with their integration services (Logic Apps, Azure Functions, Container app with Azure Application Insights).

Integration Patterns Catalogue

The section below documents some common Enterprise Integration Patterns (EIP) that can be used across different vendor platforms. The EIPs are considered industry best practices, they have been peer-reviewed by domain experts.

*Note, that this catalogue is a work in progress.

EIP Management Systems

TypeDefinition
Control Bus (CB)Manage multiple distributed integrations through a single backplane. The CB uses a separate message channel to communicate with endpoints to ensure that all integration components are working.
Alerts & NotificationAlerts and Notifications design patterns work with the Control Bus and Retry patterns. When failures occur in an integrated system, the Control Bus issues notifications to Administrators.
DetourThe Detour pattern allows a management system to send messages through a separate message channel based on conditions. This Pattern is useful for re-routing messages from the main channel.
Message HistoryEnsures that messages are persisted in storage. Commonly used in Event-driven and Event-Sourcing architectures.
Message StoreStore a Command, Event, or Query message in persistent storage for auditing purposes.
Smart Proxy
Test Message
Channel PurgerRemove all unwanted messages from a message channel.
Manual Resubmission
Failover Management
High-availability Management
Health EndpointMonitor the health of every system endpoint.
Distributed TracingTrack requests as they flow from Sender to Receiver.

Message Endpoint Patterns

TypeDefinition
Application AdapterThe Application Adapter pattern takes two incompatible interfaces and creates a bridge between the two. This pattern is very useful when converting from one message format to another (XML to JOSN).
Message Gateway
Message Mapper
Transactional Client
Polling Consumer
Event-Driven ConsumerProcess event messages based on a Sender emitting an event without the Consumer constantly polling. Event-driven EIPs notify a Consumer when a state change has occurred.
Competing ConsumerThis EIP enables multiple concurrent consumers to process messages received on the same messaging channel. With multiple concurrent consumers, a system can process multiple messages concurrently to optimize throughput, improve scalability and availability, and balance the workload.
Throttled Consumer
Singleton Consumer
Selective Consumer
Message Dispatcher
Durable Subscriber
Idempotent Receiver
Message Deduplication
Stale Message
Service Window
Service Activator
Priority QueueProcess messages based on priority queues. A Queue with a higher priority will process messages faster than a queue with a lower priority.
Queue-Based Load LevellingUse a Queue to invoke a sequential message system to manage load and costs by restricting a Consumer to processing a single message at a time.
Retry MessageA Sender and Receiver may fail to send/receive a message. Retry patterns help to solve this problem without bottlenecking the overall system.
Election LeaderA distributed system that picks the worker node based on the Bully/Ring algorithm.

Message Construction Patterns

TypeDefinition
Document Message
Event Message
Command Message
Message Model
Message Header
Request-ResponseReceive a response from a recipient after sending a message. Request-Response works well with the AMQP protocol.
Return AddressAdd Sender details to a message so that Receivers can identify and send responses back to the Sender if required.
Correlation IdentifierAdd a unique identifier that associates any message to the system or endpoint that can help to identify why the message was constructed in the first place.
Message SequenceAdd a numerical value to the message so that messages can be resequenced.
Message ExpirationAdd an expiry date on a message so that messages are automatically disposed of after a given time period.
Message Activation
Format IndicatorAdd details indicating the format of the messages so that Receivers can quickly identify and process messages.
Message SchemaCreate and transform messages from a Schema registry to ensure that a Sender & Receiver conform to a message standard.

Message Channel Patterns

TypeDefinition
Point-to-pointIntegrate a Sender & Receiver directly without using a message channel.
Publish-SubscribeProcess messages asynchronously using a message channel where Receivers consume messages when they are ready.
Pull-to-PushTransform a Pull Channel into a Push Channel.
Push-to-PullTransform a Push Channel into a Pull Channel.
DataType Channel
Invalid MessageRemove messages that contain invalid Headers/Parameters or Payload to a separate queue to process these messages independently.
Dead Letter ChannelRemove messages that cannot be delivered from a queue into a ‘Dead-Letter’ queue
Guaranteed DeliveryEnables the Sender to make sure that a message will be delivered, even if the Sender or Receiver application fails.
Message SettlementEnables a Receiver to send a response ACK to the Message Broker advising the message has been completed, abandoned, deferred or Deadlettered.
Fire & ForgetSend messages to a Receiver without expecting an acknowledgement from the Receiver.
Circuit Breaker
Channel AdapterConnect platform-agnostic systems to a single channel so that the systems can consume messages from a single channel.
Message Bus
Message BridgeConsume messages from one queue, and send them to another queue to be processed independently.
Outbox DeliveryA service that uses a relational database inserts messages or events into an outbox table (e.g. OUTBOX ) as part of the local transaction.
Delta SynchTrack database changes in one system and apply those changes to another database.

Message Transformation Patterns

TypeDefinition
Message TranslatorTranslate one message format into another.
Envelope Wrapper
Content Enricher
Content Filter
Claim CheckCall an external service that contains a larger payload than the message itself can carry.
NormalizerA message may arrive in a channel in multiple formats, XML, JSON, AVRO. Use a Normalizer EIP to transform the ingress messages to a standard format for egress.
Canonical Data Model
Service CallCall an external service to transform data held in the message.

Message Routing Patterns

TypeDefinition
Pipes-and-FiltersProcess messages through complex series of conditions and steps.
Message RouterRoute messages based on complex business rules.
Content-Based RouterRoute to a receiver based on the message header content. This design pattern works well with the AMQP protocol using the TO attribute.
Message FilterFilter messages based on rules so that Receivers only consume and process messages intended for them.
Dynamic RouterDynamically route to a destination without explicitly adding the Recipients details to a message.
Recipient ListRoute Messages to a Recipient either by using a static or dynamics Recipient list.
SplitterSplit a message into multiple channels so that they can be processed differently.
AggregatorFan-in individual messages to process them as a single collection.
ResequencerReorder a sequence of messages that have been received out of order.
Composed Msg. Processor
Scatter-Gather
Routing SlipRoute messages through a list of processing steps without predetermining the routing.
Message Broker
Process Manager
Message DelayerDelay sending a message for a given period.
Circuit BreakerStop sending messages between service endpoints if a Recipient is offline.
SagaDefine a series of related actions in a router that should be either completed successfully (all of them) or not-executed/compensated.
Service CallCall a remote service in a distributed system where the service is looked up from a service registry.
MulticastRoute single or multiple messages to multiple systems at the same time.
LoopRepeat processing the same message in a loop.
ThreadsDe-couple the routing from the current thread to a new thread to improve routing performance.
Message SamplingSample messages at intervals to a dedicated queue to ensure the overall integration is working effectively.
AbortForce message/s being delivered to a destination
Service CallCall a remote service to validate a schema or a destination address.

Comments (4)

Tony

Feb 14, 2022 at 9:36 AM

One item that’s missing is the Bi-Directional Sync. Thanks for sharing this list.

Reply

syed hussain

Jul 06, 2022 at 6:37 AM

Thank you for your comment. Integration is usually documented as a one-way stream, of course, it can be reversed so that the Receiver is also responding to a Sender making this bi-directional. If you are speaking of two-way communication (duplex) then this is a different communication protocol altogether, for this I have Azure Pub/Sub and Relays documented, but also posts on both Web Sockets and SignalR, both provide duplex capabilities.

Reply

Anton

Jun 06, 2022 at 9:32 AM

I know this page is work in progress, but where would you classify something like X12 and EDIFACT?

Reply

syed hussain

Jul 06, 2022 at 6:33 AM

I have some commentary on the Application Adapter pattern, although not explicit to EDI, it should provide some guidance.

Reply

Leave a comment