How to force schema refresh on Shared Metadata when altering delta table via Notebook

Question

How to force schema refresh on Shared Metadata when altering delta table via Notebook

Renato de Melo 125

After altering a delta table using a notebook, like adding a new field, it takes sometime for it to be reflected into Workspace Data Hub.

Documentation says it happen asynchronously, but it does not specify how long does it take.
https://learn.microsoft.com/en-us/azure/synapse-analytics/metadata/overview

Is there a way to force this synchronization?

phemanth 14,560 Reputation points Microsoft External Staff

2024-01-18T10:46:54.4266667+00:00
Thanks for reaching MS Q&A

The synchronization of changes in Azure Synapse Analytics, such as adding a new field to a delta table, is indeed an asynchronous process. The exact time it takes can vary based on several factors, including the size of the data and the current load on the system.

there are some workarounds that might help:

Incremental Load: You can set up a pipeline in Azure Data Factory that copies all tables of Azure Synapse Analytics to SQL Server dynamically. This can be set up as an incremental load.

Azure Synapse Link: If you’re using Dataverse, you can connect your data to your Azure Synapse Analytics workspace with the Azure Synapse Link service.

Table Sync: Some platforms like ThoughtSpot allow you to sync your data by adding a table sync.

please refer.

https://learn.microsoft.com/en-us/answers/questions/926206/dynamic-data-sync-between-azure-synapse-analytics

https://learn.microsoft.com/en-us/power-apps/maker/data-platform/azure-synapse-link-synapse

Please note that these methods might not result in immediate synchronization, but they can help ensure that your data is updated more frequently. Hope this helps. Do let us know if you any further queries.
phemanth 14,560 Reputation points Microsoft External Staff

2024-01-19T14:39:37.96+00:00

@Renato de Melo We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
phemanth 14,560 Reputation points Microsoft External Staff

2024-01-22T08:57:05.74+00:00

@Renato de Melo just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Renato de Melo 125 Reputation points

2024-01-22T15:53:59.54+00:00

@phemanth Thank you for your reply. I found a way to force it:

After altering the table, I just ran a SQL Script querying the table and including the new field added. Then I manually refreshed the database in the Workspace UI. After these steps, the new field appeared in the Workspace. Not sure if this will always work, but all scenarios I did that, it worked.
phemanth 14,560 Reputation points Microsoft External Staff

2024-01-23T04:16:24.5666667+00:00

@Renato de Melo
Glad to know your issue has been resolved. Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others "I'll repost your solution in case you'd like to accept the answer.
phemanth 14,560 Reputation points Microsoft External Staff

2024-01-24T08:36:54.0066667+00:00

@Renato de Melo Following up to see if the below answer was helpful. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

1 answer

Your answer

phemanth 14,560 Reputation points Microsoft External Staff

2024-01-18T10:46:54.4266667+00:00

Thanks for reaching MS Q&A

The synchronization of changes in Azure Synapse Analytics, such as adding a new field to a delta table, is indeed an asynchronous process. The exact time it takes can vary based on several factors, including the size of the data and the current load on the system.

there are some workarounds that might help:

Incremental Load: You can set up a pipeline in Azure Data Factory that copies all tables of Azure Synapse Analytics to SQL Server dynamically. This can be set up as an incremental load.

Azure Synapse Link: If you’re using Dataverse, you can connect your data to your Azure Synapse Analytics workspace with the Azure Synapse Link service.

Table Sync: Some platforms like ThoughtSpot allow you to sync your data by adding a table sync.

please refer.

https://learn.microsoft.com/en-us/answers/questions/926206/dynamic-data-sync-between-azure-synapse-analytics

https://learn.microsoft.com/en-us/power-apps/maker/data-platform/azure-synapse-link-synapse

Please note that these methods might not result in immediate synchronization, but they can help ensure that your data is updated more frequently. Hope this helps. Do let us know if you any further queries.
phemanth 14,560 Reputation points Microsoft External Staff

2024-01-19T14:39:37.96+00:00

@Renato de Melo We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
phemanth 14,560 Reputation points Microsoft External Staff

2024-01-22T08:57:05.74+00:00

@Renato de Melo just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Renato de Melo 125 Reputation points

2024-01-22T15:53:59.54+00:00

@phemanth Thank you for your reply. I found a way to force it:

After altering the table, I just ran a SQL Script querying the table and including the new field added. Then I manually refreshed the database in the Workspace UI. After these steps, the new field appeared in the Workspace. Not sure if this will always work, but all scenarios I did that, it worked.
phemanth 14,560 Reputation points Microsoft External Staff

2024-01-23T04:16:24.5666667+00:00

@Renato de Melo
Glad to know your issue has been resolved. Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others "I'll repost your solution in case you'd like to accept the answer.
phemanth 14,560 Reputation points Microsoft External Staff

2024-01-24T08:36:54.0066667+00:00

@Renato de Melo Following up to see if the below answer was helpful. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Answer 1

@Renato de Melo
Welcome to Microsoft Q&A platform and thanks for posting your question.

I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others "I'll repost your solution in case you'd like to accept the answer.

Ask: After altering a delta table using a notebook, like adding a new field, it takes sometime for it to be reflected into Workspace Data Hub.

Documentation says it happen asynchronously, but it does not specify how long does it take.
https://learn.microsoft.com/en-us/azure/synapse-analytics/metadata/overview

Is there a way to force this synchronization?

Solution: I found a way to force it:

After altering the table, I just ran a SQL Script querying the table and including the new field added. Then I manually refreshed the database in the Workspace UI. After these steps, the new field appeared in the Workspace. Not sure if this will always work, but all scenarios I did that, it worked.

If I missed anything please let me know and I'd be happy to add it to my answer, or feel free to comment below with any additional information.

If you have any other questions, please let me know. Thank you again for your time and patience throughout this issue.

---Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.

Share via

How to force schema refresh on Shared Metadata when altering delta table via Notebook

1 answer

Your answer