How to go from raw to curated in ADLS ?

Le Fur, Herve 1 Reputation point
2022-04-13T09:21:26.393+00:00

If at the raw level we have :
Raw\CRM\Customer\2022\03\06\raw_crm_customer_2022_03_06.csv
Raw\CRM\Customer\2022\03\07\raw_crm_customer_2022_03_07.csv

Do we have to keep all versions in the last level or to keep only the last view of the data ?
In case we keep only last version we will have -> Curated\CRM\Customer\crm_customer.parquet

The interest to keep a structure with no date will permit to linked directly the Synapse level to this curated space using the external object feature, isn't it ?

Or do you have another approach to manage this ?

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,397 questions
{count} votes