VS Code "Spark and Hive tools" extension failed to install Jupyter packages

Question

VS Code "Spark and Hive tools" extension failed to install Jupyter packages

Yan Xia 1

Hello

I follow the tutorials to test interactive Spark session with Synapse Spark pool in VS Code.

https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/vscode-tool-synapse

However, the "Spark and Hive Tool" extension was not able to install Jupyter package.

I have verified from that
(1) Python and Pip are installed properly.
(2) Virtualenv is installed
(3) A python environment "hdinsightJupyter" is created by under the folder .msvscode.hdinsight
(4) I can manually install Jupyter, SparkMagic and PySpark packages in the virtual environment. But the VSCode still reports Jupyter not being installed and tries to install it.
(5) I tried Python 3.7, 3.8 and 3.9. The failure is the same.

The installation failed with the error code "3221225477".

[2021-12-22T09:23:41.057] [INFO] hdiFileLogger - Jupyter not installed
[2021-12-22T09:23:41.058] [INFO] hdiFileLogger - Installing PySpark interactive virtual environment ...
[2021-12-22T09:23:41.058] [INFO] hdiFileLogger - Exec python, with args: -m,virtualenv,C:\Users\ralph.msvscode.hdinsight\hdinsightJupyter
[2021-12-22T09:23:42.111] [INFO] hdiFileLogger - created virtual environment CPython3.9.9.final.0-64 in 695ms
creator Venv(dest=C:\Users\ralph.msvscode.hdinsight\hdinsightJupyter, clear=False, no_vcs_ignore=False, global=False, describe=CPython3Windows)
seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=C:\Users\ralph\AppData\Local\pypa\virtualenv)
added seed packages: pip==21.3.1, setuptools==58.3.0, wheel==0.37.0
activators BashActivator,BatchActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

[2021-12-22T09:23:42.141] [INFO] hdiFileLogger - Exec C:\Users\ralph.msvscode.hdinsight\hdinsightJupyter\Scripts\pip.exe, with args: install,jupyter
[2021-12-22T09:23:42.569] [INFO] hdiFileLogger - Install Jupyter error:
[2021-12-22T09:23:42.569] [ERROR] hdiFileLogger - Exit with non zero 3221225477

PRADEEPCHEEKATLA 90,651 Reputation points Moderator

2021-12-23T10:51:37.763+00:00

Hello @Yan Xia ,

Thanks for the question and using MS Q&A platform.

To work with Python in Jupyter Notebooks, you must activate an Anaconda environment in VS Code, or another Python environment in which you've installed the Jupyter package. To select an environment, use the Python: Select Interpreter command from the Command Palette (Ctrl+Shift+P).

Once the appropriate environment is activated, you can create and open a Jupyter Notebook, connect to a remote Jupyter server for running code cells, and export a Jupyter Notebook as a Python file.

For more information, see Working with Jupyter Notebooks in Visual Studio Code.

1 answer

Your answer

PRADEEPCHEEKATLA 90,651 Reputation points Moderator

2021-12-23T10:51:37.763+00:00

Hello @Yan Xia ,

Thanks for the question and using MS Q&A platform.

To work with Python in Jupyter Notebooks, you must activate an Anaconda environment in VS Code, or another Python environment in which you've installed the Jupyter package. To select an environment, use the Python: Select Interpreter command from the Command Palette (Ctrl+Shift+P).

Once the appropriate environment is activated, you can create and open a Jupyter Notebook, connect to a remote Jupyter server for running code cells, and export a Jupyter Notebook as a Python file.

For more information, see Working with Jupyter Notebooks in Visual Studio Code.

Answer 1

Yan Xia 1

Thanks a lot Pradeep.

The problem is related to Synapse PySpark installation error.
I followed the same steps on Mac, and it worked.
The "Spark & Hive Tools" extension tried to install Jupyter in a specific virtual environment, hdinsightJupyter, and stuck there.

There is no details about why the installation failed.

Regards,

PRADEEPCHEEKATLA 90,651 Reputation points Moderator

2021-12-24T03:11:34.807+00:00

Hello @Yan Xia ,

Glad to know it's working on the Mac.

Could you please make sure all the prerequisites mentioned in the document are met and followed all the steps?

And also, try to do a fresh installation and see if that works.
PRADEEPCHEEKATLA 90,651 Reputation points Moderator

2021-12-28T08:33:16.403+00:00

Hello @Yan Xia ,

Follow up to see if you have had a chance to see the previous response. We need the following information to understand/investigate this issue further
Yan Xia 1 Reputation point

2022-01-04T22:06:35.517+00:00

I am sure all the prerequisites are met. The python, pip and virtualenv are installed properly.

I tried fresh installation multiple times. It is the same issue. I am using a VS Code community edition. Could it be the problem?

Thanks.
PRADEEPCHEEKATLA 90,651 Reputation points Moderator

2022-01-05T12:36:02.39+00:00

Hello @Yan Xia ,

As per the repro from my end it working as excepted on the VS Code community edition.
Yan Xia 1 Reputation point

2022-01-05T14:55:24.907+00:00

Thanks a lot Pradeep.

The task failed on executing "pip install jupyter" with error code 3221225477 very quickly. I guess it is due to a sub-process related error. Is there a way to enable debug on the command execution.

I tried launch VS Code with "code --verbose --log trace". But it does not provide debug info related to extension.

Thanks.
Yan Xia 1 Reputation point

2022-01-05T15:11:16.557+00:00

I noticed there was an error in the Windows logs under Application section. The fault application name is "python.exe" and fault module name is "ucrtbase.dll". The exception code is 5.

Any suggestions?

Thanks,
PRADEEPCHEEKATLA 90,651 Reputation points Moderator

2022-01-07T09:10:10.937+00:00

Hello @Yan Xia ,

I agree that this issue looks strange and I wasn't able to reproduce this issue. If you have a support plan could you please file a support ticket for deeper investigation.
Yan Xia 1 Reputation point

2022-01-07T14:55:11.537+00:00

Thanks. I was not able to open a support ticket under Synapse. Let me check if we have VS Code support.
PRADEEPCHEEKATLA 90,651 Reputation points Moderator

2022-01-12T07:19:53.427+00:00

Hello @Yan Xia ,

What do you mean by "I was not able to open a support ticket under Synapse"?

Share via

VS Code "Spark and Hive tools" extension failed to install Jupyter packages

1 answer

Your answer