CI CD for Synapse Spark Pool Lake Database - External Data Sources

Question

CI CD for Synapse Spark Pool Lake Database - External Data Sources

scott cooke 31

Hi, I have two Synapse environments - Dev & Prod. I have created a Lake Database and a number of tables via the Database Designer. I have setup CI/CD using the 'Deployment task for synapse workspace v2' The Lake Database and external tables deploy correctly to the Prod environment, however the external data sources deployed point to the lower (dev) environment.
After deployment if I check the tables in the Prod environment, look at the storage settings, I get a warning that the Linked Service was not found. Note that the gui is referencing the correct Linked Service for Prod. User's image

If I run

SELECT * FROM sys.external_data_sources

I can see all of the locations returned are pointing back to the dev data lake. Is there a way to parametrize the path or location of the external table. Or simply ignore in the deployment so it doesn't write over the production values.

Harishga 5,995 Reputation points Microsoft External Staff

2024-02-15T05:54:57.83+00:00

Hi @scott cooke
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Accepted answer

0 additional answers

Your answer

Harishga 5,995 Reputation points Microsoft External Staff

2024-02-15T05:54:57.83+00:00

Hi @scott cooke
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Answer 1

Hi @scott cooke

Welcome to Microsoft Q&A platform and thanks for posting your question here.

It seems that you are facing an issue with the deployment of external data sources in Azure Synapse. The problem is that the external data sources deployed point to the lower (dev) environment, causing warnings in the Prod environment.

One way to address this issue is by parametrizing the path or location of the external table. Another approach is to ignore the deployment of external data sources to prevent overwriting the production values.

Parametrize the Path: If you want to change the external data source path during deployment in Synapse Workspace v2, you can verify if the deployment task allows parameterization. This requires defining parameters for the path or location of the external table and utilizing them in the deployment process.

Ignore Deployment: Another approach is to modify the deployment process to ignore the deployment of external data sources. This would require adjusting the deployment task configuration to exclude the external data sources from the deployment process.

To fix the issue with the Linked Service not being found, you need to update the external data sources to point to the correct Linked Service in the Prod environment. You can do this by updating the external data sources using the ALTER EXTERNAL DATA SOURCE statement in SQL.

Reference
https://techcommunity.microsoft.com/t5/fasttrack-for-azure/azure-synapse-analytics-ci-cd-with-custom-parameters-made-easy/ba-p/3802517

I hope this information helps you. Let me know if you have any further questions or concerns.

scott cooke 31 Reputation points

2024-02-15T22:54:20.13+00:00

Thank you that worked. For those looking at this in the future this is what you need to add to your template-parameters-definition.json file. This will expose the Lake Database linked service and the individual tables inside.

"Microsoft.Synapse/workspaces/databases": { "properties": { "Ddls": [ { "NewEntity": { "Source": { "Location": "=" }, "StorageDescriptor": { "Format": { "Properties": { "path": "=" } }, "Source": { "Location": "=" } } } } ] }
Harishga 5,995 Reputation points Microsoft External Staff

2024-02-16T04:19:49.1633333+00:00

scott cooke
If this answers your query, please do click Accept Answer and Yes for "was this answer helpful", as it might be beneficial to other community members reading this thread. And, if you have any further query do let us know.

Share via

CI CD for Synapse Spark Pool Lake Database - External Data Sources

0 additional answers

Your answer