If you have already mastered the basics of Azure Data Factory (ADF) and you are now looking to advance your knowledge of the tool, this is the session for you. Yes, Data Factory can handle the orchestration of our ETL pipelines. But what about our wider Azure environment?
In this session we will take a deeper dive into the service, considering how to build custom activities, create metadata driven dynamic pipelines and think about hierarchical design patterns. Plus, explore ways for optimising our Azure compute costs by controlling other resource scaling as part of our normal data processing pipelines. How? Well, once we can hit a REST API from an ADF web activity anything is possible, extending our Data Factory and orchestrating everything in any data platform solution.
All this and more in a series of short lessons (based on real world experience) I will take you through how to use Azure Data Factory in production. Finally, we will look at how Data Factory can be deployed using Azure DevOps.