Differences between AMQP 1.0 vs. 0.91 Protocol

AMQP 1.0 (Advanced Message Queuing Protocol) is an open-source, binary wire protocol designed to provide a messaging backbone for distributed applications. It is designed to provide reliable, secure and robust…

MapReduce for Distributed Indexing

MapReduce is a programming model commonly used within the Hadoop framework to access big data stored in the Hadoop File System (HDFS). It consists of several phases: Map phase, Shuffle…

Point-to-Point Messaging using Azure Service Bus Queues

Here is a quick snippet of code demonstrating how Point-to-point messaging works with Azure Service Bus and the Microsoft Dataverse. The Microsoft Dataverse emits a message that is pushed into…

Ambassador Design Pattern

The Ambassador design pattern is typically used in microservice architectures, however, the same concepts can be applied to non-microservice architectures. Legacy applications that have limited features may not have the…

EIP – Application Adapter Design Pattern

Some legacy applications often communicate messages in proprietary formats, or messages the Receiver application needs to translate to make sense of. EDIFACT, X12 are common formats that often require translation/transformation.…

Event-driven Architecture using C# Observer Pattern

The Observer design pattern is a useful pattern when combined with Event-driven architectures to create reactive systems. A source system emits a state change; this change is observed by methods…

Connect to the Microsoft Dataverse via Databricks & Web API

It’s not possible to connect to the Microsoft Dataverse directly in Databricks. Some options include ingesting the Dataverse tables into Azure Data Lake and then retrieving them from inside a…

Azure Data Factory Partition Performance

There are various optimisation techniques available during the Dataflow Sink stage, in this post, I'm going to present some performance test results using various partition models. In this post, I…

Interface Definition Document

In this post, I’ve provided a sample Interface Definition Document.

Avro, Parquet, JSON, ORC Compression using ADF

The source file is a 126.48 MB CSV file which contains Postcodes and Local Authority data. In this post, I’ll be testing the file size of a transformation output to…