Find duplicate records with Azure Data Factory (same source file)

In this post I wanted to demonstrate how ADF can be used to filter out duplicate rows from a CSV file.

Save changes in Azure Data Factory with Git

Here is a quick tip. If you regularly work with Azure Data Factory, you may have noticed that you cannot save work in progress changes. This can get very tedious…

Find Duplicates using Azure Data Factory (two source files)

There are some scenarios where you need to combine different datasets with some similarities to find duplicates. In this post I demonstrate how to use Azure Data Factory and Dataflows…

Use Azure Data Factory to find duplicate records (two source files)

I've read many sources online that illustrate how to create a flow that detects record duplicates. Some of these duplicate detection patterns can be quite complex, most are old patterns.…

Stop Azure Data Factory Data Flows outputting to multiple files

I came across a recent issue where I was trying to overwrite a single in my Data Flow. However, Data Factory started to output several intermediary files instead.

Filtering in Azure Data Factory

Microsoft have now made it easier to filter records from a delimited source. We all know it's possible to filter SQL data sources, but now filtering in Azure Data Factory…

Table is required for Copy Activity

I've seen a lot of people manually create tables in Azure SQL when doing a data migration from a source system to SQL DB as the Sink. For those that…

Integration testing with Data Factory, Data Lake, Dynamics 365 Finance & Operation

Just for fun, I decided to play around with Azure Data Factory (ADF), Azure Data Lake Storage (ADLS) and Microsoft Dynamics 365 Finance & Operation VM. I wanted to see…

Designing a Modern Data Warehouse – Available Options

I recently spent some time with Microsoft to look at the various options for building our big data pipelines. Here are the final options that exist for anyone considering doing…