Azure Synapse Analytics metadata sharing

saurabh 356

Hi,

I have below questions around metadata sharing in Azure Synapse Analytic.

a) Is it possible to share metadata between spark pool and dedicated sql pool?
b) How does different spark/sql pool resource failure scenarios will affect high availability of spark/sql pool metadata?
c) What are the suggested ways to ensure high availability and disaster recovery requirements for spark/sql pool metadata?
d) Is it possible to share metadata related to delta format files between spark/sql pool and dedicated sql pool?

Please share your thoughts and any reference material to get a better understanding of above topics.

Saurabh Sharma 23,791 Reputation points Microsoft Employee

2021-08-02T21:44:32.527+00:00

Hi @saurabh ,

Thanks for using Microsoft Q&A !!
For a) No, currently it can only be shared spark pools and serverless pools.
For d) It is not possible currently and only databases and their contained external and managed tables that uses Parquet Storage format can be shared with workspace SQL engine. Please refer to
this documentation.

For b and c, could you please elaborate on what do you mean by "spark/sql pool metadata". Which all metadata objects you are referring to for high availability or for disaster recovery?
However, for c you could look for How to use CI/CD integration to automate the deploy of a Synapse Workspace to multiple environments

Thanks
Saurabh
saurabh 356 Reputation points

2021-08-03T14:19:52.823+00:00

Hi @Saurabh Sharma

Thank you for your help.

I was referring to table structure, database name, views etc.
Saurabh Sharma 23,791 Reputation points Microsoft Employee

2021-08-03T20:36:31.843+00:00

Hi @saurabh , ok but sorry but not able to understand the whole context around (b). Could you please elaborate a bit more.
Additionally for d) - Right now some information about delta table (like columns) are stored in transaction logs and but currently it is not supported syncing that.

Thanks
Saurabh
saurabh 356 Reputation points

2021-08-09T18:48:44.52+00:00

Hi @Saurabh Sharma Thanks for your reply.

The spark pool resources may become unavailable due to one or more reasons. How can one ensure HA of spark pool metadata in case an AZ/region failure occurs?

Also can stored procedure/triggers created in spark pool refer to delta tables?

Accepted answer

Saurabh Sharma 23,791 Reputation points Microsoft Employee

2021-08-11T19:06:54.027+00:00

Hi @saurabh ,
The AZ capability for Spark Databases, table definitions (plus other entities) is being already worked on it will be pushed out in all other regions probably this month end. Also, this data is already BCDR compliant.

Thanks
Saurabh
Please sign in to rate this answer.

1 person found this answer helpful.
Saurabh Sharma 23,791 Reputation points Microsoft Employee

2021-08-18T15:03:34.81+00:00

Hi @saurabh ,
Please do not forget to "Accept the answer" wherever the information provided helps you to help others in the community.

Thanks
Saurabh
Sign in to comment

Share via

Azure Synapse Analytics metadata sharing

0 additional answers