Best Practices for Creating an Identical Development Environment to Production

Question

I have a production environment with Azure Data Factory, Databricks, and Storage, and I need to replicate all of this for a development environment. What are the best practices for achieving this? I'm facing challenges with Azure Data Factory; I've tried using the ARM import and export feature, but it seems to be quite buggy. Can anyone suggest a better approach?

Answer 1

Hi Hanna,

Thanks for reaching out to Microsoft Q&A.

Use Infrastructure as Code (IaC)

• Leverage Infrastructure as Code tools like Azure Resource Manager (ARM) templates or terraform to define and provision your development environment infrastructure.

Automate Environment Provisioning

• Automate the provisioning of your development environment using CI/CD pipelines.This ensures that every time you need to create or update your development environment, it's done consistently and reliably.

Use Linked Services for External Dependencies

• When configuring Azure Data Factory, use Linked Services to connect to external resources like Databricks and Storage.

Separate Configuration from Code

• Store environment-specific configuration, such as connection strings and keys, separately from your Data Factory code.

Regularly Refresh Development Data

• Ensure that your development environment contains realistic data by regularly refreshing it from production backups or snapshots.

This helps catch data-related issues early in the development process and ensures that your code works as expected with real-world data. By following these best practices, you can create a reliable and consistent development environment that can closely mirror your prod setup and making it easier to develop, test and deploy changes.

Please 'Upvote'(Thumbs-up) and 'Accept' as an answer if the reply was helpful. This will also help us close this thread and acknowledge the time spent by community volunteers like us.

Answer 2

Hi Hanna ,

Thankyou for using Microsoft Q&A platform and thanks for posting your query here.

As per my understanding you are trying to replicate the prod environment to dev environment for ADF, databricks and storage. However, that's not actually how it works.

Deployment is always supposed to go from Dev to higher environment, and never backwards.

To get the details around the implementation of Continuous integration and delivery , kindly check out the documentation : Continuous integration and delivery in Azure Data Factory

However, if it's one time replication from prod to dev , then you can consider cloning the environment . Kindly check out the documentation for : Copy or clone a data factory in Azure Data Factory

Hope it helps. Kindly accept the answer by clicking on Accept answer button. Thankyou