Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,920 questions
How to use the whl file in the Spark pool in Azure Synapse Analytics?
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(51.2, 69%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESK%3C/text%3E%3C/svg%3E)
Sanket Kelkar
0
Reputation points
Hi All,
I want to use a .whl file in the Spark Pool of Azure Synapse Analytics. There are total 3 ways that I have tried -
A. From the Azure Portal - by manually adding the .whl file to the workspace packages and then to the spark pool packages. This method is too slow and takes approx. 30 mins to complete.
B. From the Azure CLI (in Powershell) - by running the below commands -
- Upload the .whl file to the workspace
az synapse workspace-package upload --workspace-name myWorkSpace --package "C:\Users\Test\Documents\GitHub\TestGitHub\dist\my_etl-0.0.1-py3-none-any.whl"
- Attach the package to the spark pool
az synapse spark pool update --name mySparkPoolName--workspace-name myWorkSpace --resource-group myRG --package-action Add --package my_etl-0.0.1-py3-none-any.whl
This method is also slow and takes approx. 20 mins to complete.
C. From the Storage account that is linked to the Spark pool -
I uploaded the .whl file to the location -
/mySynapseLinkedStorage/synapse/workspaces/myWorkSpace/sparkpools/mySparkPool/libraries/python/my_etl-0.0.1-py3-none-any.whl
My question is, how do I link this file which is in my storage to the Spark Pool?
Also, is there any faster option to attach the .whl file to the Spark pool?
Thank you!
Sanket Kelkar
{count} votes
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 26%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HimanshuSinha-msft 19,476 Reputation points Microsoft Employee2023-03-30T21:11:16.2033333+00:00 Hello @Sanket Kelkar , Thanks for the question and using MS Q&A platform.
My understanding is since you are already passing the path below which had the sparkpool name , next time when you restart the Spark Pool (in this case mySparkPool) , you should have the library installed.
/mySynapseLinkedStorage/synapse/workspaces/myWorkSpace/sparkpools/mySparkPool/libraries/python/my_etl-0.0.1-py3-none-any.whl
Please do let me know if you are seeing anything different.
Thanks Himanshu
Please accept as "Yes" if the answer provided is useful , so that you can help others in the community looking for remediation for similar issues.
-
Sanket Kelkar 0 Reputation points
2023-04-04T10:10:38.4533333+00:00 Hi , No, when the spark pool is restarted, it still can't find the .whl package. One observation - I was able to get it working using the Spark Pool built on Apache Spark 2.4. But it fails for Apache Spark 3.1, 3.2, and 3.3. Do you know if MS supports the .whl file installation in this fashion for Apache Spark 3.x? Thank you!
Sanket Kelkar
-
Sanket Kelkar 0 Reputation points
2023-04-11T08:32:34.5366667+00:00 Hi @HimanshuSinha-msft @HimanshuSinha-msft
Please let me know your thoughts on the above point. I am stuck on this and can't proceed with Azure Synpase (using Apache Spark 3.3 version).
Sign in to comment
Sign in to answer