EIP – Durable Subscriber

Summary

Right, let’s dive into the world of Durable Subscribers and how they help us avoid losing messages like forgotten emails in an overloaded inbox. Picture this scenario: you’ve got a system humming along, sending out messages like there’s no tomorrow, and all is well. But then—your subscriber goes offline. Does that mean messages are lost in the ether, never to be seen again? Absolutely not, if you’re using a Durable Subscriber.

At its core, the Durable Subscriber pattern is about making sure that any messages meant for a temporarily disconnected subscriber don’t vanish into thin air. Instead, they’re stored, waiting patiently for the subscriber to reconnect. Let’s break down how this works, how you can implement it, and why you might want to use it.

Why Do We Need Durable Subscribers?

In any messaging system, there’s a sender (the publisher) and one or more receivers (subscribers). The publisher sends messages into some kind of message broker—think of it as a postal service—and the subscribers pick them up. But what happens when a subscriber is unavailable? Maybe it’s down for maintenance, or maybe it’s just briefly disconnected. In non-durable systems, any message sent during this period would be lost, like a letter sent to an old address.

This is where durable subscriptions come into play. They make sure that the messages still get delivered—even if the subscriber isn’t immediately available. The message broker stores them until the subscriber reconnects, and once it’s back online, all the saved messages are delivered. Simple, right? But it’s also a game changer for systems where you absolutely cannot afford to lose any messages.

Here’s a basic rundown of what happens in a Durable Subscriber system:

1. The publisher sends messages to a message broker.
2. The broker checks whether the subscriber is online.
3. If the subscriber is online, the message is delivered immediately.
4. If the subscriber is offline, the message is saved (or queued) in the broker.
5. Once the subscriber reconnects, the broker sends the saved messages.

In essence, this pattern is all about reliability. It guarantees message delivery, no matter the availability status of the subscriber.

The Message Broker: The Middleman of Messaging

The magic behind durable subscriptions lies in the message broker. It’s the broker’s job to receive messages from publishers and deliver them to subscribers. If a subscriber goes offline, the broker holds onto the messages until the subscriber comes back. Several messaging platforms provide this functionality, but let’s look at a couple of the popular options to see how they handle durable subscriptions.

Apache Kafka

Kafka is known for its durability and reliability, so naturally, it supports durable subscribers. In Kafka, messages are stored in topics. When a subscriber (often called a consumer in Kafka’s terminology) reads from a topic, it keeps track of its position in the message stream (called an offset). If the subscriber disconnects, Kafka holds onto all new messages until the subscriber reconnects and picks up where it left off.

Kafka is designed to handle massive volumes of data, so durable subscriptions are baked into its core functionality. Messages are never lost because they’re persisted to disk, meaning Kafka can keep hold of them for days or even weeks, depending on your configuration.

Azure Service Bus

Another popular option is Azure Service Bus, which offers built-in support for durable subscribers through its topic and subscription model. A topic in Service Bus is like a post office where messages are sent, and subscriptions are like the mailboxes for individual subscribers. If a subscriber is temporarily unavailable, Service Bus stores the messages in the subscription until the subscriber comes back online.

What’s especially nice about Service Bus is that it allows you to define message retention policies, so you can control how long messages are stored. And because Service Bus is fully managed, you don’t have to worry about the underlying infrastructure—Microsoft handles it for you.

Both Kafka and Azure Service Bus excel at durable subscriptions, but they aren’t the only options. RabbitMQ, AWS SNS (Simple Notification Service), and others also offer similar functionality. The key point is that whichever platform you choose, the broker is responsible for holding onto messages and ensuring they get delivered.

How Durable Subscriptions Work

Let’s go a bit deeper into how this works under the hood. The key components of a durable subscription system are:

1. The Publisher: This is the entity that sends messages into the system. It could be a web app, an IoT device, a service, or anything that needs to send data somewhere else.
2. The Message Broker: This is where all the heavy lifting happens. It acts as an intermediary between publishers and subscribers, ensuring that messages are delivered to the right place. The broker is responsible for saving messages when subscribers go offline and sending them when they reconnect.
3. The Subscriber: This is the recipient of the messages. Subscribers typically register their interest in receiving messages from a specific topic or queue. The key characteristic of a durable subscriber is that it can be offline and still receive all messages that were sent during its downtime.
4. The Subscription: This is the actual link between the subscriber and the topic or queue. In some systems, like Azure Service Bus, the subscription is what keeps track of which messages have been delivered and which are waiting to be sent. In Kafka, the subscription is tied to the consumer group, which tracks the offsets (i.e., where the consumer left off).

When the publisher sends a message, the broker checks to see if the subscriber is connected. If it is, the message is delivered right away. If not, the broker stores the message in a queue or topic. When the subscriber comes back online, the broker retrieves all stored messages and sends them to the subscriber in the order they were received.

This ensures that no messages are lost, even if the subscriber is offline for an extended period. However, the details of how this works vary between platforms, so let’s take a closer look at Kafka and Azure Service Bus.

Durable Subscriptions in Apache Kafka

In Kafka, durability is built-in by design. When a message is published to a Kafka topic, it is written to disk and replicated across multiple brokers in the cluster. This makes it incredibly reliable. Subscribers in Kafka are called consumers, and they can be grouped into consumer groups.

How Kafka Handles Durability:

1. Offset Tracking: Each consumer in Kafka keeps track of the last message it read, which is referred to as the offset. If the consumer disconnects, Kafka doesn’t throw away the unread messages—it simply stores them in the topic. When the consumer reconnects, it resumes from the last recorded offset.
2. Retention Policies: Kafka topics can be configured with a retention period, which determines how long messages are stored. Even if a subscriber doesn’t reconnect for hours (or even days), the messages will be waiting for it.
3. Partitioning: Kafka divides topics into partitions, which allows for parallel processing of messages. Each consumer in a group is assigned a partition, and Kafka ensures that the offset is tracked for each partition independently. This means that consumers can process messages independently, and if one consumer goes down, the others can continue processing their partitions.

Key Configuration Points in Kafka:

• Enable Group Rebalancing: When a consumer goes offline, Kafka can redistribute the load among the remaining consumers in the group, ensuring that messages continue to be processed even if one consumer is temporarily unavailable.
• Retention Periods: Configure topic retention to ensure messages are kept long enough for subscribers to come back online. You don’t want messages to expire before a subscriber reconnects.

Durable Subscriptions in Azure Service Bus

Azure Service Bus offers durable subscriptions through its topic and subscription model. The key here is that each subscriber has its own subscription, which stores messages until the subscriber is ready to process them.

How Azure Service Bus Handles Durability:

1. Topic-Subscription Model: When a message is sent to a topic, it is delivered to all subscribers that have an active subscription. If a subscriber is offline, the messages are held in the subscription queue until the subscriber reconnects.
2. Message Retention: Service Bus allows you to configure how long messages are kept in a subscription. You can set the retention period to match your needs, whether that’s a few minutes or several days. This ensures that messages aren’t discarded too soon.
3. Dead-letter Queues: If a subscriber fails to process a message within a certain time frame, it can be moved to a dead-letter queue, which is a special holding place for messages that couldn’t be delivered. This ensures that messages aren’t lost, even if they can’t be processed right away.

Key Configuration Points in Azure Service Bus:

• Auto-forwarding: If a subscriber doesn’t reconnect within a set timeframe, messages can be automatically forwarded to another topic or queue, ensuring they aren’t lost.
• Lock Duration: You can configure how long a message stays locked to a subscriber before it becomes available to other subscribers.

Handling Message Ordering and Duplicates

One of the challenges with durable subscribers is ensuring that messages are delivered in the correct order, especially if a subscriber is offline for a while. Most messaging platforms, including Kafka and Azure Service Bus, handle this by storing messages in the order they were received. However, depending on your setup, there are a few things to watch out for:

1. Message Ordering: In Kafka, messages are stored in partitions, and each partition guarantees message order. In Azure Service Bus, you can use sessions to ensure that messages within the same session are delivered in order.
2. Duplicate Messages: Sometimes, subscribers might receive duplicate messages, especially if they disconnect and reconnect quickly. It’s essential to design your system to handle duplicate messages gracefully. This can be done by assigning unique message IDs and having the subscriber check whether it has already processed a message before acting on it.

Advantages of Durable Subscribers

So why bother with durable subscribers in the first place? Here are a few compelling reasons:

1. Reliability: Durable subscribers guarantee that messages won’t be lost, even if a subscriber goes offline. This is crucial for systems that rely on timely and accurate message delivery.
2. Fault Tolerance: If one part of your system goes down, the rest of the system can continue functioning without interruption. When the subscriber comes back online, it can pick up where it left off.
3. Scalability: Durable subscriptions allow you to scale your system without worrying about losing messages. You can add or remove subscribers without affecting message delivery.
4. Flexibility: Durable subscribers can reconnect at any time and still receive all messages that were sent while they were offline. This is ideal for distributed systems where components might go up and down unpredictably.

Full CLI Commands Example

To sum it all up, here’s a streamlined set of Azure CLI commands that would create the entire setup:

# Step 1: Create a resource group
az group create --name myResourceGroup --location eastus

# Step 2: Create a Service Bus namespace
az servicebus namespace create --resource-group myResourceGroup --name myServiceBusNamespace --location eastus --sku Standard

# Step 3: Create a queue
az servicebus queue create --resource-group myResourceGroup --namespace-name myServiceBusNamespace --name myQueue --max-size 1024

# Optional: Create a topic for pub/sub scenario
az servicebus topic create --resource-group myResourceGroup --namespace-name myServiceBusNamespace --name myTopic

# Optional: Create a subscription to the topic
az servicebus topic subscription create --resource-group myResourceGroup --namespace-name myServiceBusNamespace --topic-name myTopic --name mySubscription

Conclusion

The Durable Subscriber pattern is essential for ensuring that no messages are lost when a subscriber is temporarily unavailable. By leveraging durable subscriptions, message brokers like Kafka and Azure Service Bus can hold onto messages until the subscriber reconnects. Whether you’re dealing with a microservices architecture, IoT devices, or any system where reliability matters, durable subscribers provide peace of mind by guaranteeing message delivery.

In short, if you’re working with any kind of messaging system where subscribers can’t always be online, durable subscriptions are the way to go. They save messages, ensure reliability, and make sure your system keeps ticking along smoothly, no matter what happens to individual components.