Implementing CI/CD for Azure Data Factory
Send me Nitor Infotech's Monthly Blog Newsletter!
×
nitor logo
  • Company
    • About
    • Leadership
    • Partnership
  • Resource Hub
  • Blog
  • Contact
nitor logo
Add more content here...
Artificial intelligence Big Data Blockchain and IoT
Business Intelligence Careers Cloud and DevOps
Digital Transformation Healthcare IT Manufacturing
Mobility Product Modernization Software Engineering
Thought Leadership
Aastha Sinha Abhijeet Shah Abhishek Suranglikar
Abhishek Tanwade Abhishek Tiwari Ajinkya Pathak
Amit Pawade Ankita Kulkarni Ankita Patidar
Antara Datta Anup Manekar Ashish Baldota
Chandra Gosetty Deep Shikha Bhat Dr. Girish Shinde
Ekta Shah Gaurav Mishra Gaurav Rathod
Gautam Patil Harish Singh Chauhan Harshali Chandgadkar
Kapil Joshi Krishna Gunjal Madhavi Pawar
Marappa Reddy Mayur Wankhade Milan Pansuriya
Minal Doiphode Mohit Agarwal Mohit Borse
Nalini Vijayraghavan Nikhil Kulkarni Omkar Ingawale
Omkar Kulkarni Pooja Chavan Pooja Dhule
Pranit Gangurde Prashant Kankokar Priya Patole
Rahul Ganorkar Rashmi Nehete Ravi Agrawal
Robin Pandita Rohan Chavan Rohini Wwagh
Sachin Saini Sadhana Sharma Sambid Pradhan
Sandeep Mali Sanjay Toge Sanjeev Fadnavis
Saurabh Pimpalkar Sayanti Shrivastava Shardul Gurjar
Shravani Dhavale Shreyash Bhoyar Shubham Kamble
Shubham Muneshwar Shubham Navale Shweta Chinchore
Sidhant Naveria Souvik Adhikary Sujay Hamane
Tejbahadur Singh Uddhav Dandale Vasishtha Ingale
Vidisha Chirmulay Yogesh Kulkarni
Cloud and DevOps | 22 May 2023 |   7 min

Implementing CI/CD for Azure Data Factory

featured image

Continuous Integration (CI) is a practice which allows developers to seamlessly merge code in a common repository. Continuous Delivery (CD), on the other hand, is a practice that adds a layer to CI by providing multi-stage infrastructure provisioning and deployment which helps in automating the entire software release process.

For Azure data pipelines, CI/CD means nothing but moving data factory pipelines from one environment to another.

In this blog, I will walk you through the process of implementing continuous integration and continuous delivery for pipelines created in Azure Data Factory (ADF).

To begin with, let us first take a look at the high-level architecture:

ADF High Level Architecture

From this, we can concur that the ADF is used to perform ETL/ELT operations on data. To implement CI/CD in ADF, Azure DevOps repository needs to be configured first. As shown in above flow, once the pipeline is developed and published, changes can be pushed in the master branch through Pull Request following which a Release pipeline is created to deploy these changes on Test and Production environments. This is done by configuring environment specific connections, which is elaborated later.

Before we move any further, let me tell you that an Azure DevOps account is a mandatory stepping stone if you want to successfully implement CI/CD in ADF.

Now, since implementing CI/CD is the main target of this blog, we will create and use a simple ADF pipeline.

Here are the steps for it:

1. Configure Azure DevOps repository in ADF:

First and foremost, we will create an ADF pipeline in Dev environment and then will deploy it in a QA environment through CI/CD process.

i. Create resource group for Dev environment:

Implementing CI_CD with ADF 01

ii. Create ADF instance under Dev resource group:

Implementing CI_CD with ADF 02

iii. Create resource group for QA environment

Implementing CI_CD with ADF 03

iv. Create an ADF instance on QA environment:

Implementing CI_CD with ADF 04

v. Now, create new project in Azure DevOps as shown in the diagram below:

Implementing CI_CD with ADF 05

vi. Configure code repository in ADF:

Implementing CI_CD with ADF 06

vii. Configure repository as shown in the diagram below:

Implementing CI_CD with ADF 07

viii. Select a working branch:

Implementing CI_CD with ADF 08

ix. Add a pipeline to the ADF. These changes will be saved in the above branch.

Implementing CI_CD with ADF 09

x. Create a Pull request to merge the above branch in master branch:

Implementing CI_CD with ADF 10

xi. Once this Pull Request is approved, changes will be merged into master branch:

Implementing CI_CD with ADF 11

xii. ADF pipeline changes are seen under master branch:

Implementing CI_CD with ADF 12

Step 2: Creating Release Pipeline in Azure DevOps:

i. Go to Pipeline section in Azure DevOps:

Implementing CI_CD with ADF 13

ii. Fill in the details as shown below:

Implementing CI_CD with ADF 14

iii. Add and configure steps in Agent Job:

Implementing CI_CD with ADF 15

iv. Configure the pipeline as mentioned in below diagram:

Implementing CI_CD with ADF 16

v. For configuring templates, click on the eclipse button and select ArmTemplateforFactory json file.

Implementing CI_CD with ADF 17

vi. For configuring template parameter section, click on the eclipse button and select ArmParameterTemplateforFactory json file.

Implementing CI_CD with ADF 18

vii. Parameter override section:

Provide the name required in QA environment ADF instance.

viii. In order to enable continuous integration, enable below option in trigger section.

Implementing CI_CD with ADF 19

ix. Once completed, click on Save button:

Implementing CI_CD with ADF 20

x. Click on run to build artefacts of the pipeline:

Implementing CI_CD with ADF 21

Step 3: Creating the final release:

i. Click on new Pipeline:

Implementing CI_CD with ADF 22

ii. Once clicked on new Pipeline, configure the Release pipeline as shown below:

Implementing CI_CD with ADF 23

iii. In stage section add QA instance name:

Implementing CI_CD with ADF 24

iv. Once configured, click on add as well as on continuous deployment trigger symbol to enable.

v. Now click on configure Stage section -> Job link:

Implementing CI_CD with ADF 25

Once this is created, it will add release in queue and will deploy the changes on ADF QA environment.

You can now make changes in Dev environment ADF instance and the same will be released on QA environment ADF instance.

There you have it. Now, that wasn’t so difficult was it?

Reach out to us at Nitor Infotech if you want to learn more about our DevOps offering and how we successfully implemented DevOps for a leading product development company to reduce their defect creation time by 14%.

Related Topics

Artificial intelligence

Big Data

Blockchain and IoT

Business Intelligence

Careers

Cloud and DevOps

Digital Transformation

Healthcare IT

Manufacturing

Mobility

Product Modernization

Software Engineering

Thought Leadership

<< Previous Blog fav Next Blog >>
author image

Nitor Infotech Blog

Nitor Infotech is a leading software product development firm serving ISVs and enterprise customers globally.

   

You may also like

featured image

15 Performance Improvement Techniques for Your iOS App

In the world of iOS app development, app performance refers to the speed, responsiveness, and ...
Read Blog


featured image

The Ultimate Guide to Different Types of Testing

In today’s competitive scenario, businesses that want to stand out against their peers must invest in building best-in-class software that is performant and failure-proof. To ensure sustained funct...
Read Blog


featured image

The Importance of ChatGPT and Why it is Becoming Popular

Imagine having a conversation with a chatbot that feels almost human. That’s exactly what OpenAI ChatGPT brings to the table. The remarkable technology of Generative Pre-trained Transformer (GPT) p...
Read Blog


subscribe

Subscribe to our fortnightly newsletter!

We'll keep you in the loop with everything that's trending in the tech world.

Services

    Modern Software Engineering


  • Idea to MVP
  • Quality Engineering
  • Product Engineering
  • Product Modernization
  • Reliability Engineering
  • Product Maintenance

    Enterprise Solution Engineering


  • Idea to MVP
  • Strategy & Consulting
  • Enterprise Architecture & Digital Platforms
  • Solution Engineering
  • Enterprise Cognition Engineering

    Digital Experience Engineering


  • UX Engineering
  • Content Engineering
  • Peer Product Management
  • RaaS
  • Mobility Engineering

    Technology Engineering


  • Cloud Engineering
  • Cognitive Engineering
  • Blockchain Engineering
  • Data Engineering
  • IoT Engineering

    Industries


  • Healthcare
  • Retail
  • Manufacturing
  • BFSI
  • Supply Chain

    Company


  • About
  • Leadership
  • Partnership
  • Contact Us

    Resource Hub


  • White papers
  • Brochures
  • Case studies
  • Datasheet

    Explore More


  • Blog
  • Career
  • Events
  • Press Releases
  • QnA

About


With more than 16 years of experience in handling multiple technology projects across industries, Nitor Infotech has gained strong expertise in areas of technology consulting, solutioning, and product engineering. With a team of 700+ technology experts, we help leading ISVs and Enterprises with modern-day products and top-notch services through our tech-driven approach. Digitization being our key strategy, we digitally assess their operational capabilities in order to achieve our customer's end- goals.

Get in Touch


  • +1 (224) 265-7110
  • marketing@nitorinfotech.com

We are Social 24/7


© 2023 Nitor Infotech All rights reserved

  • Terms of Usage
  • Privacy Policy
  • Cookie Policy
We use cookies to ensure that we give you the best experience on our website. If you continue to use this site we will assume that you are happy with it. Accept Cookie policy