Delta Lake Data lake design

Question

Hi friends we are designing data lake and three stages raw ,silver and gold my question is do we need to have staging layer also,what are the advantages and is it compulsory

Answer 1

Hello @Anshal ,

Thanks for the question and using MS Q&A platform.

Data lake synonyms in use Raw layer = staging = bronze = landing zone.

The Bronze layer is where we land all the data from source systems. The table structures in this layer correspond to the source system table structures “as-is,” aside from optional metadata columns that can be added to capture the load date/time, process ID, etc.

In most cases, it's a good idea to keep the data in the Bronze layer in Delta format, so that subsequent reads from the Bronze layer for ETL are performant — and so that you can do updates in Bronze to write CDC changes. Sometimes, when data arrives in JSON or XML formats, we do see customers landing it in the original source data format and then stage it by changing it to Delta format. So sometimes, we see customers manifest the logical Bronze layer into a physical landing and staging zone.

It's important to plan your data structure before you land it into a data lake. Having a plan helps you use security, partitioning, and processing effectively.

For more details, refer to Data lake zones and containers.

Hope this will help. Please let us know if any further queries.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is jhow you can be part of Q&A Volunteer Moderators