How to arrange private packages for use on ADX

Question

How to arrange private packages for use on ADX

Villarreal, Andrea L 41

Regarding using external artifacts in the python plugin in Azure Data Explorer

The documentation includes these two related notes about private python packages and wheels:

For private packages, zip the folder of the package and the folders of its dependencies.
Make sure to zip the .whl files themselves, and not their parent folder

My question is, what does this look like in practice? Say that I have the following private python package:

my_private_pkg.whl

and it requires the following python packages not available in the ADX python plugin:

python_package_not_avail_on_ADX_1.whl

python_package_not_avail_on_ADX_2.whl

How should this all be organized and uploaded to my storage container? I've managed to have some success, but it seems like my approach is hacked together, and likely not the most efficient way.

Andrea

PRADEEPCHEEKATLA 91,496 Reputation points Moderator

2023-06-19T06:02:10.11+00:00
@Villarreal, Andrea L - Thanks for the question and using MS Q&A platfrom.

To use private Python packages on ADX, you need to upload the required packages to your storage container and then install them on the ADX cluster. Here's how you can organize and upload your private Python packages:

Create a folder named "packages" in your storage container.

Inside the "packages" folder, create a folder named "my_private_pkg".

Copy the "my_private_pkg.whl" file to the "my_private_pkg" folder.

Create a folder named "dependencies" inside the "my_private_pkg" folder.

Inside the "dependencies" folder, create folders for each of the required packages, i.e., "python_package_not_avail_on_ADX_1" and "python_package_not_avail_on_ADX_2".

Copy the ".whl" files for each of the required packages to their respective folder

Your folder structure should look like this:

packages/my_private_pkg.zip └── my_private_pkg/ ├── my_private_pkg.whl └── dependencies/ ├── python_package_not_avail_on_ADX_1/ │ └── python_package_not_avail_on_ADX_1.whl └── python_package_not_avail_on_ADX_2/ └── python_package_not_avail_on_ADX_2.whl

Then, when you install the packages on the ADX cluster, you can specify the location of the zip files.

For more details, refer to https://learn.microsoft.com/en-us/azure/data-explorer/kusto/query/pythonplugin?pivots=azuredataexplorer#install-packages-for-the-python-plugin.

Hope this helps. Do let us know if you any further queries.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Villarreal, Andrea L 41

Hi Pradeep,

I have built the zip as specified in your answer. How do I then make use of what is in my_private_pkg?

I replicated the environment setup on an IDE by creating a python virtual env, and then pip installed the my_private_pkg wheel. The syntax in this setup for running a function from my_private_pkg, once the python interpreter is activated, is:

import Utils.event_detection.my_py_file

foo = my_py_file.my_function(...kwargs...)

However, this same syntax gets an error in ADX's python plugin:


Error 

 Query execution has resulted in error (0x80131500): Partial query failure: 0x80131500 (message: 'Encountered an error during execution of local sandbox. Error details: Python code execution failed with the following error: ModuleNotFoundError: No module named 'Utils'; Traceback (most recent call last):
  File "C:\python\execute_python.py", line 164, in exec_python
    exec(code_to_execute, local_variables, local_variables)
  File "<string>", line 10, in <module>
ModuleNotFoundError: No module named 'Utils'
.  ==> ExecutePluginOperator failure: ', details: 'Source: DataNode
[0]Kusto.Cloud.Platform.Sandbox.Exceptions.SandboxExecutionException: Encountered an error during execution of local sandbox. Error details: Python code execution failed with the following error: ModuleNotFoundError: No module named 'Utils'; Traceback (most recent call last):
  File "C:\python\execute_python.py", line 164, in exec_python
    exec(code_to_execute, local_variables, local_variables)
  File "<string>", line 10, in <module>
ModuleNotFoundError: No module named 'Utils'

...


ClientRequestId=
ActivityId=00000000-0000-0000-0000-000000000000
Details=Python code execution failed with the following error: ModuleNotFoundError: No module named 'Utils'; Traceback (most recent call last):
  File "C:\python\execute_python.py", line 164, in exec_python
    exec(code_to_execute, local_variables, local_variables)
  File "<string>", line 10, in <module>
ModuleNotFoundError: No module named 'Utils'


   at Kusto.DataNode.QueryService.PluginsV2.SandboxedPluginBase.ExecuteInSandbox(SandboxKind sandboxKind, ISandboxManager sandboxManager, ClientRequestProperties clientRequestProperties, IDictionary`2 argumentsPropertyBag, IStreamSource inputStreamSource, IDictionary`2 externalArtifacts, Boolean spillToDisk, OperationStatistics& operationStatistics) in C:\__w\1\s\Src\Engine\DataNode\QueryService\PluginsV2\DistributedPlugins\SandboxedPluginBase.cs:line 61
   at Kusto.DataNode.QueryService.PluginsV2.ScriptExecutionPluginBase.Execute(QueryPluginExecutionContext executionContext, PluginDistributionCapsule distributionCapsule, ClientRequestProperties clientRequestProperties, IStreamSource inputStreamSource, OperationStatistics& operationStatistics) in C:\__w\1\s\Src\Engine\DataNode\QueryService\PluginsV2\DistributedPlugins\Languages\ScriptExecutionPluginBase.cs:line 164
   at Kusto.DataNode.DataEngineQueryPlan.DataEngineQueryProcessor.DataEngineQueryCallback.ExecutePluginOperator(String pluginName, DataSourceStreamFormat inputStreamFormat, DataSourceStreamFormat outputStreamFormat, String pluginSerializedContext, String serializedQueryContextProperties, IStreamSource inputStream, OperationStatistics& operationStatistics) in C:\__w\1\s\Src\Engine\DataNode\QueryService\DataEngineQueryPlan\DataEngineQueryProcessor.cs:line 460').

 clientRequestId: KustoWebV2;767a1b3e-b1d7-4291-954e-9d6ffe1fcf0e

PRADEEPCHEEKATLA 91,496 Reputation points Moderator

2023-06-22T06:03:29.0666667+00:00

@Villarreal, Andrea L - We are reaching out to the internal team to get more information related to your query and will get back to you as soon as we have an update.
PRADEEPCHEEKATLA 91,496 Reputation points Moderator

2023-06-23T04:55:56.4133333+00:00

@Villarreal, Andrea L - You should create a zip file name it as you like, e.g. my_package.zip, containing my_private_pkg.whl, python_package_not_avail_on_ADX_1.whl & python_package_not_avail_on_ADX_2.whl. When you call Zipackage.install() it starts by unzipping the artifact to a temporary directory in the sandbox, then unzipping each wheel and finaly add that temp directory to Python modules search path.

Hope this helps. Do let us know if you any further queries.
Villarreal, Andrea L 41 Reputation points

2023-06-23T15:05:18.4+00:00
Hi Pradeep,

I tested this today:

Zip file format:

my_package.zip └── my_private_pkg.whl └── python_package_not_avail_on_ADX_1.whl └── python_package_not_avail_on_ADX_2.whl

ADX Python Plugin actions with succeed/fail notes:

Zipackage.install("my_package.zip")

import my_private_pkg # Result: ModuleNotFoundError

import submodule_in_my_private_pkg # Result: ModuleNotFoundError

On another note, possibly related, I have been getting a similar error when trying to install a pypi publicly available package, called ruptures.

Not sure if it would be more appropriate for me to submit a support ticket on the Azure portal? However, I do not have the proper permissions within my organization to do so.
PRADEEPCHEEKATLA 91,496 Reputation points Moderator

2023-06-26T06:15:40.8166667+00:00

@Villarreal, Andrea L - Thanks for the details. We are reaching out to the internal team to get more information related to your query and will get back to you as soon as we have an update. details,

Your answer

PRADEEPCHEEKATLA 91,496 Reputation points Moderator

2023-06-19T06:02:10.11+00:00

@Villarreal, Andrea L - Thanks for the question and using MS Q&A platfrom.

To use private Python packages on ADX, you need to upload the required packages to your storage container and then install them on the ADX cluster. Here's how you can organize and upload your private Python packages:

Create a folder named "packages" in your storage container.

Inside the "packages" folder, create a folder named "my_private_pkg".

Copy the "my_private_pkg.whl" file to the "my_private_pkg" folder.

Create a folder named "dependencies" inside the "my_private_pkg" folder.

Inside the "dependencies" folder, create folders for each of the required packages, i.e., "python_package_not_avail_on_ADX_1" and "python_package_not_avail_on_ADX_2".

Copy the ".whl" files for each of the required packages to their respective folder

Your folder structure should look like this:

packages/my_private_pkg.zip └── my_private_pkg/ ├── my_private_pkg.whl └── dependencies/ ├── python_package_not_avail_on_ADX_1/ │ └── python_package_not_avail_on_ADX_1.whl └── python_package_not_avail_on_ADX_2/ └── python_package_not_avail_on_ADX_2.whl

Then, when you install the packages on the ADX cluster, you can specify the location of the zip files.

For more details, refer to https://learn.microsoft.com/en-us/azure/data-explorer/kusto/query/pythonplugin?pivots=azuredataexplorer#install-packages-for-the-python-plugin.

Hope this helps. Do let us know if you any further queries.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
PRADEEPCHEEKATLA 91,496 Reputation points Moderator

2023-06-22T06:03:29.0666667+00:00

@Villarreal, Andrea L - We are reaching out to the internal team to get more information related to your query and will get back to you as soon as we have an update.
PRADEEPCHEEKATLA 91,496 Reputation points Moderator

2023-06-23T04:55:56.4133333+00:00

@Villarreal, Andrea L - You should create a zip file name it as you like, e.g. my_package.zip, containing my_private_pkg.whl, python_package_not_avail_on_ADX_1.whl & python_package_not_avail_on_ADX_2.whl. When you call Zipackage.install() it starts by unzipping the artifact to a temporary directory in the sandbox, then unzipping each wheel and finaly add that temp directory to Python modules search path.

Hope this helps. Do let us know if you any further queries.
Villarreal, Andrea L 41 Reputation points

2023-06-23T15:05:18.4+00:00

Hi Pradeep,

I tested this today:

Zip file format:

my_package.zip └── my_private_pkg.whl └── python_package_not_avail_on_ADX_1.whl └── python_package_not_avail_on_ADX_2.whl

ADX Python Plugin actions with succeed/fail notes:

Zipackage.install("my_package.zip")

import my_private_pkg # Result: ModuleNotFoundError

import submodule_in_my_private_pkg # Result: ModuleNotFoundError

On another note, possibly related, I have been getting a similar error when trying to install a pypi publicly available package, called ruptures.

Not sure if it would be more appropriate for me to submit a support ticket on the Azure portal? However, I do not have the proper permissions within my organization to do so.
PRADEEPCHEEKATLA 91,496 Reputation points Moderator

2023-06-26T06:15:40.8166667+00:00

@Villarreal, Andrea L - Thanks for the details. We are reaching out to the internal team to get more information related to your query and will get back to you as soon as we have an update. details,

Share via

How to arrange private packages for use on ADX

Your answer