Azure datalake creation methods

Question

Hi friends, we need to create a datalake (ADLS) with bronze, silver, and gold layers. One of our colleagues suggested it is a better idea to create ADLS through programming, but we plan to create it manually. Are there any advantages of each approach and what is the general and most used approach?

Answer 1

Hello Anshal,

Thank you for posting your query in the Microsoft Q&A Community.

Please be informed that approach depends on your organization’s best practices. Most Organization use both. Manual setup are used during initial development and testing (It’s an opportunity to learn about ADLS features, security, and best practices.) while programmatic setup are used for consistency, scalability, and automation.

You can integrate ADLS setup into your deployment pipelines (e.g., Azure DevOps, Terraform, ARM templates when you leverage on automation

Let me know if further assistance is needed.

Babafemi

Answer 2

@Anshal

Manual approach advantages:
You get direct control over the creation and configuration of your ADLS resources, which can be beneficial for understanding the setup and configuration. Manual creation can be a good learning experience, especially if you're new to Azure and its ecosystem.

Programmatic Approach advantages:
Programmatic approaches (using Azure CLI, PowerShell, or ARM templates) allow for automated and repeatable deployments, which is ideal for consistency and scalability. More efficient for large-scale projects as it can handle complex structures and large volumes of data. Easier to integrate with CI/CD pipelines and other automated workflows.

The most common approach in the industry is to use programming for creating and managing ADLS. This is because it offers greater flexibility, scalability, and efficiency, especially for larger projects. However, a manual approach might be simpler for small-scale or one-time setups.