Data Lake Organization Pattern

Question

Data Lake Organization Pattern

grajee 371

What are the ways in which a data lake can be organized? This is in reference to the Zones and access.

First Option:
RawZone (Container)\Retail\Full
RawZone (Container)\Retail\Incr

StageZone (Container)\Retail\Full
StageZone (Container)\Retail\Incr
CuratedZone (Container)\Retail\Global

RawZone (Container)\Wholesale\Full
RawZone (Container)\ Wholesale\Incr
RawZone (Container)\ Wholesale\ITD

StageZone (Container)\Wholesale\Full
StageZone (Container)\Wholesale\Incr

CuratedZone (Container)\WholeSale\Global

Second Option:
Retail(Container)\RawZone\Full
Retail(Container)\RawZone\Incr
Retail(Container)\RawZone\ITD
Retail(Container)\StageZone\Full
Retail(Container)\GoldZone\Global

Wholesale(Container)\RawZone\Full
Wholesale(Container)\RawZone\Incr
Wholesale(Container)\RawZone\ITD
Wholesale(Container)\GoldZone\Global

SubjectAreas are– Retail, Wholesale, Online, Lease … Rentals.

In the first option, data files will be organized Starting with the Zones
In the second option, data files will be organized Starting with the subject area

The second option is very specific to a SubjectArea and the access rights can be easily assigned to specific groups but the issue is there are many RawZone as many subject areas (same applies to other zones as well). In the first approach, applying RBAC at the Zone level results in the rights getting inherited to the folders resulting in users not belonging to Retail getting access to WholeSale which means we have to use ACLs (?) to remove the unwanted permissions.

Thanks,
grajee

HimanshuSinha 19,637 Reputation points Microsoft Employee Moderator

2021-06-08T23:33:16.617+00:00

Hello @grajee ,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .In case if you have any resolution please do share that same with the community as it can be helpful to others . Otherwise, will respond back with the more details and we will try to help .
Thanks
Himanshu
HimanshuSinha 19,637 Reputation points Microsoft Employee Moderator

2021-06-09T18:28:54.573+00:00

Hello @grajee ,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .In case if you have any resolution please do share that same with the community as it can be helpful to others . Otherwise, will respond back with the more details and we will try to help .
Thanks
Himanshu

1 answer

Your answer

HimanshuSinha 19,637 Reputation points Microsoft Employee Moderator

2021-06-08T23:33:16.617+00:00

Hello @grajee ,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .In case if you have any resolution please do share that same with the community as it can be helpful to others . Otherwise, will respond back with the more details and we will try to help .
Thanks
Himanshu
HimanshuSinha 19,637 Reputation points Microsoft Employee Moderator

2021-06-09T18:28:54.573+00:00

Hello @grajee ,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .In case if you have any resolution please do share that same with the community as it can be helpful to others . Otherwise, will respond back with the more details and we will try to help .
Thanks
Himanshu

Answer 1

Hello @grajee ,
Thanks for the ask and using the Microsoft Q&A platform .
Let me acknowledge that your ask was very descriptive , and I understand the ask here is to use RBAC or ACL ? Since you are looking for granular permission I could suggest you to go with the ACL . After looking at the folder structure it appears that you will have to manually set the ACL which can be a challenge for so many folders .Not sure if you have already gone through the doc here , sharing it just in case .
.
Thanks
Himanshu
Please do consider clicking on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members

Share via

Data Lake Organization Pattern

1 answer

Your answer