Git Configuration for Azure Data Factory: Deploying Existing Branches on New Environments
Send me Nitor Infotech's Monthly Blog Newsletter!
×
nitor logo
  • Company
    • About
    • Leadership
    • Partnership
  • Resource Hub
  • Blog
  • Contact
nitor logo
Add more content here...
Artificial intelligence Big Data Blockchain and IoT
Business Intelligence Careers Cloud and DevOps
Digital Transformation Healthcare IT Manufacturing
Mobility Product Modernization Software Engineering
Thought Leadership
Aastha Sinha Abhijeet Shah Abhishek Suranglikar
Abhishek Tanwade Abhishek Tiwari Ajinkya Pathak
Amit Pawade Ankita Kulkarni Ankita Patidar
Antara Datta Anup Manekar Ashish Baldota
Chandra Gosetty Deep Shikha Bhat Dr. Girish Shinde
Ekta Shah Gaurav Mishra Gaurav Rathod
Gautam Patil Harish Singh Chauhan Harshali Chandgadkar
Kapil Joshi Krishna Gunjal Madhavi Pawar
Marappa Reddy Mayur Wankhade Milan Pansuriya
Minal Doiphode Mohit Agarwal Mohit Borse
Nalini Vijayraghavan Nikhil Kulkarni Omkar Ingawale
Omkar Kulkarni Pooja Chavan Pooja Dhule
Pranit Gangurde Prashant Kankokar Priya Patole
Rahul Ganorkar Rashmi Nehete Ravi Agrawal
Robin Pandita Rohan Chavan Rohini Wwagh
Sachin Saini Sadhana Sharma Sambid Pradhan
Sandeep Mali Sanjay Toge Sanjeev Fadnavis
Saurabh Pimpalkar Sayanti Shrivastava Shardul Gurjar
Shravani Dhavale Shreyash Bhoyar Shubham Kamble
Shubham Muneshwar Shubham Navale Shweta Chinchore
Sidhant Naveria Souvik Adhikary Sujay Hamane
Tejbahadur Singh Uddhav Dandale Vasishtha Ingale
Vidisha Chirmulay Yogesh Kulkarni
Big Data | 26 Apr 2023 |   8 min

Git Configuration for Azure Data Factory: Deploying Existing Branches on New Environments

featured image

Imagine a scenario wherein you are making drastic changes in your existing data factory. This could be by changing naming conventions or handing over your development resources to a new customer. There are several instances where you may need to create a new data factory. This may be to develop in, or at least revise the source of your existing data factory and bring in a new repo.

All of this, and then some, can be done with the help of Azure Data Factory’s Git configuration. With it, you can readily manage the contents of your existing data factory with no hassle at all.

In my blog, I will show you how you can deploy an existing data factory repository brand on a new environment with the help of this Git configuration.

But, before I begin, I want to shed light on a prerequisite that will help you with this migration. You must create environment-specific resources (such as Global Parameter, Private End Point, Integration Runtime) on the new Data Factory environment. This is so that when we point it to our branch, the existing manually created resources will be published automatically to the branch.

An added benefit of this is that it resolves the error “Pipeline is not found, please publish first.”

Now, without further ado, here is how you do it.

1. Consider the branch “release” to be deployed on Azure Data Factory through Git Configuration. This branch will not have “factory” and “managedVirtualNetwork” folders. This is because these will be created manually on the new environment and automatically in this branch once pointed to the new Data Factory environment.

Deploy existing data factory repository brand on a new environment via Git configuration 1 Nitor Infotech

Fig 1: The branch “release” to be deployed on Azure Data Factory through Git Configuration

2. Go to Data Factory environment and Open Git Configuration (Manage -> Git Configuration).

Deploy existing data factory repository brand on a new environment via Git configuration 2 Nitor Infotech

Fig 2: Open Git Configuration

3. Click on Configure and follow the steps as shown below.

Deploy existing data factory repository brand on a new environment via Git configuration 3 Nitor Infotech

Fig 3: Configure a repository

4. Set the “release” branch as Collaboration branch. Tick the “Import existing resources to repository” option as given below. Then click on Apply.

Deploy existing data factory repository brand on a new environment via Git configuration 4 Nitor Infotech

Fig 4: Configure a repository

5. After clicking on Apply, the already existing resources will be added/published in the “release” branch. We had created these manually on Data Factory before connecting it to the “release” branch.

This happened because we checked the “Import existing resources to repository” option during Git configuration.

So, we will get two new folders in the release branch as shown below:

Deploy existing data factory repository brand on a new environment via Git configuration 5 Nitor Infotech

Fig 5: Two new folders in the release branch

6. Now go to Azure DevOps and create a local branch from the “release” branch with any name. This new local branch is for temporary use.

In our case, we named it as “ProdRepo”.

Deploy existing data factory repository brand on a new environment via Git configuration 6 Nitor Infotech

Fig 6: Create a local branch

7. The “ProdRepo” branch will be a replica of the “release” branch as shown below:

Deploy existing data factory repository brand on a new environment via Git configuration 7 Nitor Infotech

Fig 7: The “ProdRepo” branch – a replica of the “release” branch

8. Now go to Data Factory and disconnect the existing branch “release” from Data Factory.

Deploy existing data factory repository brand on a new environment via Git configuration 8 Nitor Infotech

Fig 8: Disconnect the existing branch “release” from Data Factory

9. Follow steps 2 to 4 as mentioned above. The exception is that you need to uncheck the “Import existing resources to repository” option and set the Collaboration branch as “ProdRepo”.

Deploy existing data factory repository brand on a new environment via Git configuration 9 Nitor Infotech

Fig 9: Set the Collaboration branch as “ProdRepo”

10. After Clicking on Apply, your Data Factory will point to the new temporary branch i.e., “ProdRepo”.

Then Click on Publish to deploy the branch on the Data Factory.

Deploy existing data factory repository brand on a new environment via Git configuration 10 Nitor Infotech

Fig 10: Publish to deploy the branch on the Data Factory

  • As we know, the “adf_publish” branch is by default a publish branch in ADF and here we have the “ProdRepo” branch as a Collaboration Branch.
  • Azure wants to get deployed everything on the adf_publish branch from Collaboration Branch “ProdRepo”. This is so that Data Factory can run independently even without pointing to any Collaboration branch.
  • After publishing now, we can still see all the pipelines, datasets, linked services, and other resources on ADF. This is even if we disconnect the Collaboration branch (through Git Configuration of data Factory),
  • This is because everything has been published on adf_branch now and does not need a Collaboration_Branch anymore i.e., ProdRepo.
  • Publishing everything helps you to resolve the error – “Pipeline is not found, please publish first.”

11. After successfully Publishing, now we can disconnect (as shown in Step No. 8) the “ProdRepo” Collaboration branch. We can again point the “release” Collaboration branch (as shown in Steps 2, 3 and 4). This is except for unchecking the “Import existing resources to repository”.

And there you have it! You are now equipped to move your existing data factory repositories to new ones using the Git configuration as arsenal.

Write to us at Nitor Infotech. Learn more about how you can manage copious amounts of data with our data engineering services.

Related Topics

Artificial intelligence

Big Data

Blockchain and IoT

Business Intelligence

Careers

Cloud and DevOps

Digital Transformation

Healthcare IT

Manufacturing

Mobility

Product Modernization

Software Engineering

Thought Leadership

<< Previous Blog fav Next Blog >>
author image

Robin Pandita

Senior Software Engineer

Robin is an experienced module lead and a BI developer with more than 5 years of working experience in the information technology and services industry. He is skilled in ETL tools like Azure Data Factory and Pentaho DI, business reporting tools like Power BI, and familiar with basics of Tableau, as well as database tools like SQL server and Oracle SQL. Additionally, he also has a good understanding of Data Warehouse and data modelling concepts. In his spare time, he loves to sing and play guitar. He abides by his self-made thought- “When your ability weighs more than probability, there lies no chance of impossibility”.

   

You may also like

featured image

15 Performance Improvement Techniques for Your iOS App

In the world of iOS app development, app performance refers to the speed, responsiveness, and ...
Read Blog


featured image

The Ultimate Guide to Different Types of Testing

In today’s competitive scenario, businesses that want to stand out against their peers must invest in building best-in-class software that is performant and failure-proof. To ensure sustained funct...
Read Blog


featured image

The Importance of ChatGPT and Why it is Becoming Popular

Imagine having a conversation with a chatbot that feels almost human. That’s exactly what OpenAI ChatGPT brings to the table. The remarkable technology of Generative Pre-trained Transformer (GPT) p...
Read Blog


subscribe

Subscribe to our fortnightly newsletter!

We'll keep you in the loop with everything that's trending in the tech world.

Services

    Modern Software Engineering


  • Idea to MVP
  • Quality Engineering
  • Product Engineering
  • Product Modernization
  • Reliability Engineering
  • Product Maintenance

    Enterprise Solution Engineering


  • Idea to MVP
  • Strategy & Consulting
  • Enterprise Architecture & Digital Platforms
  • Solution Engineering
  • Enterprise Cognition Engineering

    Digital Experience Engineering


  • UX Engineering
  • Content Engineering
  • Peer Product Management
  • RaaS
  • Mobility Engineering

    Technology Engineering


  • Cloud Engineering
  • Cognitive Engineering
  • Blockchain Engineering
  • Data Engineering
  • IoT Engineering

    Industries


  • Healthcare
  • Retail
  • Manufacturing
  • BFSI
  • Supply Chain

    Company


  • About
  • Leadership
  • Partnership
  • Contact Us

    Resource Hub


  • White papers
  • Brochures
  • Case studies
  • Datasheet

    Explore More


  • Blog
  • Career
  • Events
  • Press Releases
  • QnA

About


With more than 16 years of experience in handling multiple technology projects across industries, Nitor Infotech has gained strong expertise in areas of technology consulting, solutioning, and product engineering. With a team of 700+ technology experts, we help leading ISVs and Enterprises with modern-day products and top-notch services through our tech-driven approach. Digitization being our key strategy, we digitally assess their operational capabilities in order to achieve our customer's end- goals.

Get in Touch


  • +1 (224) 265-7110
  • marketing@nitorinfotech.com

We are Social 24/7


© 2023 Nitor Infotech All rights reserved

  • Terms of Usage
  • Privacy Policy
  • Cookie Policy
We use cookies to ensure that we give you the best experience on our website. If you continue to use this site we will assume that you are happy with it. Accept Cookie policy