Data Lake Design Pattern

Question

Data Lake Design Pattern

saurabh 361

Hi ,

In the below documentation on "Overview of Azure Data Lake Storage for the data management and analytics scenario", its mentioned that one should have one resource group for 3 ADLS accounts. Is it implied that as part of the recommended design pattern, one should use the same set of three ADLS accounts for both Prod and Non-Prod?

https://learn.microsoft.com/en-us/azure/cloud-adoption-framework/scenarios/data-management/best-practices/data-lake-overview

Does the recommended design pattern prefers to have different resource groups for data integration, data lake and data products(data processing) in different subscriptions or same subscription? Or will such an implementation considered an anti pattern?

Accepted answer

0 additional answers

Your answer

Answer 1

KranthiPakala-MSFT 46,642 Microsoft Employee Moderator

Hello @saurabh ,

Thanks for the question and using MS Q&A platform.

As per my understanding, you would like to know (also please find my answers inline):

As part of recommended design pattern from the documentation you referred, one should use the same set of three ADLS accounts for both Prod and Non-Prod?

Yes, we are proposing the same structure across all environments (dev/test/prod). A storage account does not create any cost if not used and hence there is no reason to change the setup in dev or pre-prod. In general, one should try to make the various environments as similar as possible. Otherwise, we will always end up with various kinds of problems once you move the code into higher environments. Always try to have an exact replica in lower environments. For cost reasons you might want to scale down some services, but that is not applicable to these storage accounts.

Does the recommended design pattern prefers to have different resource groups for data integration, data lake and data products (data processing) in different subscriptions or same subscription?

As per the design pattern shown here or here Data Lakes, Data Products and Data Integration Products will land in the Data Landing Zone into different resource groups within same subscription. Data Lakes have their own resource group. There are a couple of core resources in each landing zone to create a metadata driven ingestion framework (ADF, Azure SQL, Key Vault, SHIR), which are also landing in a few resource groups in the core landing zone setup.

Data Products are then environments where cross-functional data teams can work on their data use cases. These use cases get their own resource group, subnet and space on the data lakes to implement their own solution. Inside that resource group they are free to create the services they require within the boundaries of Azure Policy. They are free to choose between Synapse, ADF, AML, etc. or they can use some of the shared Databricks instances that get deployed with each landing zone.

Hope this info helps.

------------------------------

Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
Want a reminder to come back and check responses? Here is how to subscribe to a notification
If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

saurabh 361 Reputation points

2022-01-29T06:13:58.423+00:00

@KranthiPakala-MSFT Thanks for taking time to answer. From the first part of your answer, I understand that the recommendation is to have different storage accounts for Prod and Non-Prod rather than use the same storage accounts for both prod and non-prod data.

Also do you think in this architecture pattern, the prod and non-prod resource groups should be hosted in same subscription or different subscriptions (say 1 subscription for prod, 1 for QA and 1 for dev, rather than using same subscription for prod/qa/dev)?
KranthiPakala-MSFT 46,642 Reputation points Microsoft Employee Moderator

2022-02-02T00:33:53.833+00:00

Hey @saurabh , thanks for getting back.

Our internal solution experts are recommending to land the dev/test/prod instances of the landing zones in separate subscriptions. The data integration products should follow this pattern and can use the same respective subscriptions. The data products however often have to land their dev/test/prod instances into the prod landing zone as they have to access prod data in each of their environment. Development without prod data is often a problem and copying the data into the dev landing zone is not a recommended approach.

Hope this info helps.

----------

Please do consider to click on "Accept Answer" and "Upvote" on the post that helps you, as it can be beneficial to other community members.
saurabh 361 Reputation points

2022-02-02T02:34:27.427+00:00

@KranthiPakala-MSFT Thanks for your reply.

Development without prod data is often a problem and copying the data into the dev landing zone is not a recommended approach.

Do you think using a data masking process to desensitize the prod data before copying to dev could be a good option and mitigate the data quality issues for development ? Generally speaking making prod data available as is for development might not meet an organizations security requirements.
KranthiPakala-MSFT 46,642 Reputation points Microsoft Employee Moderator

2022-02-02T06:52:12.5+00:00

In general, creating data copies is not a good idea and something that most organizations don't want to do. Copies need to be maintained and need to be updated, which is why most customers rely on in-place sharing. Copying prod data into a dev environment is something that most organizations won't do.
saurabh 361 Reputation points

2022-02-02T15:43:13.827+00:00

Thanks for your prompt reply and all the help. Lets say if one application needs to modify the data, and we want to test that application in development environment. With the suggested approach, should people be allowed to work on production copy of the data with a risk of potentially corrupting the data ? What benefits one can get by masking data in same prod environment and not mask copy in another environment ?
KranthiPakala-MSFT 46,642 Reputation points Microsoft Employee Moderator

2022-02-04T01:10:10.603+00:00

Hello @saurabh ,

In general, our internal experts are proposing to use an in-place data access model. The data source will be read-only by data consumers and modifications cannot be made to the source. The teams can read the data, process the data and store their own results in their dedicated workspace file system, as required. In summary, the source cannot be overwritten and, hence, there is no risk of corrupting the data. But ultimately the decision to choose which data to be used for development testing will depends on the sensitivity/confidentiality of the data as well as organizations security requirements.

Hope this helps. If the original query of this thread was answered, we recommend opening a new thread on further topics so that it will have high visibility to the broader community.

Thank you

Share via

Data Lake Design Pattern

0 additional answers

Your answer