Adding whl files to an Azure Synapse spark pool

Question

According to the documentation, we should be able to add custom libraries as follows:

az synapse spark pool update --name testpool \  
--workspace-name testsynapseworkspace --resource-group rg \  
--package-action Add --package package1.jar package2.jar  

However, when I try this with my python package whl files, I get an error message that the package does not exist.

> $new_package_names = "PACKAGE1-1.0.1-py3-none-any.whl PACKAGE2-1.0.6.3-py3-none-any.whl PACKAGE3-1.0.0-py3-none-any.whl PACKAGE4-1.0.1-py3-none-any.whl"    
  
> az synapse spark pool update --name $pool_name --workspace-name $workspace_name --resource-group $resource_group --package-action Add --package $new_package_names  

 

I receive the following error since two or three weeks ago (before that it worked flawlessly):

(LibraryDoesNotExistInWorkspace) The LibraryArtifact PACKAGE1-1.0.1-py3-none-any.whl PACKAGE2-1.0.6.3-py3-none-any.whl PACKAGE3-1.0.0-py3-none-any.whl PACKAGE4-1.0.1-py3-none-any.whl does not exist.  
Code: LibraryDoesNotExistInWorkspace  
Message: The LibraryArtifact PACKAGE1-1.0.1-py3-none-any.whl PACKAGE2-1.0.6.3-py3-none-any.whl PACKAGE3-1.0.0-py3-none-any.whl PACKAGE4-1.0.1-py3-none-any.whl does not exist.  

The same works if I have only one package in the variable $new_package_names. It looks to me like Azure thinks it's all one package instead of four different ones. All four are uploaded to the synapse workspace and available for selection when I do the same process manually. Adding the packages manually also works without a problem.

Since I receive this error only since 2 or 3 weeks and it worked before without a problem, I was wondering whether there was an update that I missed? However, the latest documentation still describes the same procedure.

Answer 1

Hello @Victor I have some useful information for you.

As I was leaving work yesterday I was chatting with my colleague about this issue.

If you are using az in powershell, there is a better way to go about this.

 $new_package_names = "PACKAGE1-1.0.1-py3-none-any.whl" , "PACKAGE2-1.0.6.3-py3-none-any.whl" , "PACKAGE3-1.0.0-py3-none-any.whl" , "PACKAGE4-1.0.1-py3-none-any.whl"    
      
 az synapse spark pool update --name $pool_name --workspace-name $workspace_name --resource-group $resource_group --package-action Add --package @new_package_names  

Here we changed new_package_names into an array type, and use the @ splatter operator to seperate them.

As simpler example, it makes the following two excerpts be equivalent:

Copy-Item "test.txt" "test2.txt" -WhatIf  
  
$ArrayArguments = "test.txt", "test2.txt"  
Copy-Item @ArrayArguments -WhatIf  

Hope this will help. Please let us know if any further queries.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators