Adding whl packages failing on DEP Enabled workspace.

Question

Hello,

I am having some installation errors when trying to download great expectations and google-analytics data whl packages and the dependencies in my DEP enabled workspace.

I have followed this guide on installing whl packages Install third party whl packages I made some slight alterations since my spark pool is using Apache runtime 3.3. I used the latest miniforge3 shell since it comes with python=3.10 as the base and loaded a virtual environment with the Synapse-python310.yml file

I attached a picture of the whl packages and dependencies I generated for both great expectations and google analytics data.

This is the error i get when loading all the 19 whl packages.

ERROR: Wheel 'jsonpointer' located at /usr/lib/library-manager/bin/lmjob/w2prdimdsynapse/wheels/jsonpointer-2.4-py2.py3-none-any.whl is invalid.

What i don't seem to understand is, I have a non-DEP enabled workspace, and when i put both great expectations and google-analytics-data in the requirements.txt file. The same dependencies install completely fine, according to the sttdout logs.

How can I address this error?

Answer 1

Hi Brian Maina ,

Thankyou for putting effort and sharing the approach for resolving the issue.

Glad that you figured out the way to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to "Accept " the answer.

Uploading the wheel package to blob storage using blob fuse and retrieving them using Azure Storage Explorer caused changes in the MD5 hashes of the packages. By using drag and drop functionality within xrdp helped resolve the issue.

Kindly accept the answer by clicking on Accept answer button so that the solution reaches the community in case anyone is facing similar issue. Thankyou.

Answer 2

I found the issue. I was making the packages within my Ubuntu instance Azure VM and load them to a blob storage via blob fuse. When i downloaded the packages from the blob using Azure storage explorer, it would change the MD5 hashes of the packages.

I found out that i can drag and drop files within xrdp and avoid using blobfuse to transfer files outside the VM