Post by: syed hussain in All Azure Data Factory Reference Architecture
There are various optimisation techniques available during the Dataflow Sink stage, in this post, I'm going to present some performance test results using various partition models. In this post, I…
Post by: syed hussain in All Azure Data Factory
Azure Data Factory has a great set of tools to manage data pipelines. One of those tools is the ability to create arrays and use the array properties as parameters.…
Post by: syed hussain in All Azure Data Factory
In this post I wanted to demonstrate how ADF can be used to filter out duplicate rows from a CSV file.
Post by: syed hussain in Azure Data Factory
Here is a quick tip. If you regularly work with Azure Data Factory, you may have noticed that you cannot save work in progress changes. This can get very tedious…
Post by: syed hussain in All Azure Data Factory
There are some scenarios where you need to combine different datasets with some similarities to find duplicates. In this post I demonstrate how to use Azure Data Factory and Dataflows…
Post by: syed hussain in Azure Data Factory
I've read many sources online that illustrate how to create a flow that detects record duplicates. Some of these duplicate detection patterns can be quite complex, most are old patterns.…
Post by: syed hussain in Azure Data Factory
I came across a recent issue where I was trying to overwrite a single in my Data Flow. However, Data Factory started to output several intermediary files instead.
Post by: syed hussain in Azure Data Factory
Microsoft have now made it easier to filter records from a delimited source. We all know it's possible to filter SQL data sources, but now filtering in Azure Data Factory…
Post by: syed hussain in Azure Data Factory
I've seen a lot of people manually create tables in Azure SQL when doing a data migration from a source system to SQL DB as the Sink. For those that…
Just for fun, I decided to play around with Azure Data Factory (ADF), Azure Data Lake Storage (ADLS) and Microsoft Dynamics 365 Finance & Operation VM. I wanted to see…