Issue with copy data from Azure Synapse Link for Dataverse using ADF Dataflow

Raheel Islam 21 Reputation points
2024-07-01T11:07:06.5533333+00:00

Hi there,

We use Export to Data Lake option to copy the data from D365 and want to transform to Azure Synapse Link for Dataverse because Export to Data Lake option is obsoleting from 1st Nov 2024.

We have configured Azure Synapse link for Dataverse in UAT environment as per below screenshot. User's image

It has created the new container with different timestamp folders.

User's image

Above folders contain tables and data information.

User's image

I have created a data pipeline in ADF to get the custtable data using dataflow and give the path till highlighted in above screenshot. It works fine till here but as I make changes in D365, a new folder with different timestamp created with the data and modify rows only as can see in below screenshot.

User's image

I want to get custtable or any specific entity data using ADF Dataflow but here the issue is I need to define the full path till timestamp and this timestamp folder create whenever any update happen in the entity, so this is not the fix path. How I get the custtable data everyday but how to know the path?

here is the Dataflow code:

User's image

User's image

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,416 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,073 questions
{count} votes