How to install wheel file on synapse sparkpool during deployment

Sanjeev Nayak 1 Reputation point
2022-06-29T14:43:03.947+00:00

Hi,
I have created a wheel file to manage my adhoc libraries and to use some custom functions to reuse my python notebook. How to install it on spark pool?, I don't want to install manually, going to the package then upload and attach to the sparkpool or through requrements.txt file. How to avoid manual interference.

Also observed after each deployment the sparkpool getting over written, that means the wheel file you need to install again after each deployment manually.

I checked in sparkpool 2.4 version you upload the wheelfile into the libraries/Python it automatically get installed but its having lot of problem w.r.t to existing libarries on the sparkpool, many commands is not working on 2.4 version sparkpool.

Please provide me a solution on this, either to upload somewhere it will automatically installed with the latest version sparkpool or during deployment through yml/CICD it can be handled or any other solution to automate it, please let me know.

any solution to this would be highly appreciated.

Regards
Sanjeev

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,422 questions
0 comments

2 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. ShaikMaheer-MSFT 37,971 Reputation points Microsoft Employee
    2022-07-01T10:38:49.817+00:00

    Hi @Sanjeev Nayak ,

    Thank you for posting query in Microsoft Q&A Platform.

    The manual process you describe can probably be automated in an Azure DevOps pipeline, though there is no magic easy button. Instead of uploading the files as workspace packages, upload them to a storage account and reference them in the job definition, but the general idea is the same. Have custom scripts using either the Synapse Azure CLI commands or Az.Synapse PowerShell cmdlets to update the job definitions with the storage paths to the packages.

    Kindly try and see if that helps. Please let us know how it goes. Thank you.

    0 comments
  2. Sanjeev Nayak 1 Reputation point
    2022-07-06T14:28:49.06+00:00

    Hi,
    I am able to install the wheel file on sparkpool cluster by using below powershell command

    $package = New-AzSynapseWorkspacePackage -WorkspaceName workspace1 -Package "C:\xxx-2.0.2-py3.whl"
    Update-AzSynapseSparkPool -WorkspaceName workspace1 -Name pool2 -PackageAction Add -Package $package

    Hope this would help.