How to install wheel file on synapse sparkpool during deployment

Sanjeev Nayak 1 Reputation point

I have created a wheel file to manage my adhoc libraries and to use some custom functions to reuse my python notebook. How to install it on spark pool?, I don't want to install manually, going to the package then upload and attach to the sparkpool or through requrements.txt file. How to avoid manual interference.

Also observed after each deployment the sparkpool getting over written, that means the wheel file you need to install again after each deployment manually.

I checked in sparkpool 2.4 version you upload the wheelfile into the libraries/Python it automatically get installed but its having lot of problem w.r.t to existing libarries on the sparkpool, many commands is not working on 2.4 version sparkpool.

Please provide me a solution on this, either to upload somewhere it will automatically installed with the latest version sparkpool or during deployment through yml/CICD it can be handled or any other solution to automate it, please let me know.

any solution to this would be highly appreciated.


Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,328 questions
0 comments No comments
{count} votes

2 answers

Sort by: Most helpful
  1. ShaikMaheer-MSFT 37,656 Reputation points Microsoft Employee

    Hi @Sanjeev Nayak ,

    Thank you for posting query in Microsoft Q&A Platform.

    The manual process you describe can probably be automated in an Azure DevOps pipeline, though there is no magic easy button. Instead of uploading the files as workspace packages, upload them to a storage account and reference them in the job definition, but the general idea is the same. Have custom scripts using either the Synapse Azure CLI commands or Az.Synapse PowerShell cmdlets to update the job definitions with the storage paths to the packages.

    Kindly try and see if that helps. Please let us know how it goes. Thank you.

    0 comments No comments

  2. Sanjeev Nayak 1 Reputation point

    I am able to install the wheel file on sparkpool cluster by using below powershell command

    $package = New-AzSynapseWorkspacePackage -WorkspaceName workspace1 -Package "C:\xxx-2.0.2-py3.whl"
    Update-AzSynapseSparkPool -WorkspaceName workspace1 -Name pool2 -PackageAction Add -Package $package

    Hope this would help.