Is there a way to dynamically send data sets via Azure Data Share only once the DWH has completed?

DataPeps2023 25

Having had our first DWH (data warehouse) failure since setting up Azure data share for about 5 external customers, I'm now wondering if there is a way for the data share to wait for the DWH to complete / wait for a trigger?

1 answer

QuantumCache 20,261 Reputation points

2023-10-03T23:44:55.6+00:00
Hello @DataPeps2023

Did you try to consider Azure Data Factory: By using ADF to orchestrate the data movement, you can ensure that the data is moved to Azure Data Share only when the data warehouse has completed, and you can move only the changed data to minimize the data movement time and cost.

ADF pipeline that moves data from your data warehouse to Azure Data Share.

use the appropriate ADF triggers to trigger the pipeline based on a schedule or an event.

Configure the pipeline to move only the data that has changed since the last run. You can use the appropriate ADF data flow transformations to compare the data in the data warehouse with the data in Azure Data Share and move only the changed data.
Please sign in to rate this answer.
DataPeps2023 25 Reputation points

2023-10-04T10:58:03.5866667+00:00

Thanks for your response. I hadn't appreciated that ADF could orchastrate the movement to Azure Data Share.

It would have to go via Azure Synapse to get to the data share?

Is that our data share or the customer's data share?
1. If our data share, how does it trigger to push to the customer?

QuantumCache 20,261 Reputation points

2023-10-09T23:48:26.58+00:00

@DataPeps2023

The data would need to go through Azure Synapse Analytics if you are sharing data from your Synapse workspace to your data share. If you are sharing data from another supported data store, the data would need to go through that data store.

If you are the data provider, you can create a snapshot schedule that allows data consumers to receive regular, incremental updates to the data shared with them. When a data consumer accepts and configures their data share, they can subscribe to a snapshot schedule. This is beneficial in scenarios where the shared data is updated regularly, and the data consumer needs the most up-to-date data.

I hope this helps with your initial query, please let us know if we can close this escalation?

DataPeps2023 25 Reputation points

2023-10-10T08:45:20.7566667+00:00

Hi thanks again for your response. That makes sense. However, at the moment, we are sending from Azure SQL Database and only daily, as the data warehouse only updates once a day, in the morning. I don't think snapshots are available for SQL Database.

QuantumCache 20,261 Reputation points

2023-10-12T18:29:49.19+00:00

Hi @DataPeps2023 This is a great discussion topic, i hope useful to other members of the community.

Please let me know if we can close this case?
Sign in to comment

Share via

Is there a way to dynamically send data sets via Azure Data Share only once the DWH has completed?

1 answer