question

anadelcampo avatar image
0 Votes"
anadelcampo asked SanjeevNayak-3936 commented

Synapse Analytics Requirements.txt not loaded

I have been trying to install google-api's libraries to a notebook, I need to extract google classroom data into synapse. I uploaded the requirements.txt file - even created a new cluster. Not the libraries are not being loaded, even after re-starting the cluster. I have reviewed other posts with similar questions, followed the instructions but still cannot install the libraries. Here is the entire requirements file - did a pip freeze and added the needed libraries at the bottom:

absl-py==0.8.1
adal==1.2.2
alabaster==0.7.10
altair==3.2.0
applicationinsights==0.11.9
asn1crypto==1.0.1
astor==0.8.0
astroid==1.4.9
astropy==1.3.2
attrs==19.2.0
azure-common==1.1.25
azure-core==1.8.1
azure-graphrbac==0.61.1
azure-identity==1.2.0
azure-mgmt-authorization==0.60.0
azure-mgmt-containerregistry==2.8.0
azure-mgmt-keyvault==2.0.0
azure-mgmt-resource==5.1.0
azure-mgmt-storage==4.2.0
azure-storage-blob==2.1.0
azure-storage-common==2.1.0
azureml-automl-core==1.10.0.post1
azureml-automl-runtime==1.10.0.post1
azureml-core==1.10.0.post1
azureml-dataprep==1.10.1
azureml-dataprep-native==14.2.1
azureml-dataset-runtime==1.10.0
azureml-defaults==1.10.0
azureml-explain-model==1.10.0
azureml-interpret==1.10.0
azureml-model-management-sdk==1.0.1b6.post1
azureml-opendatasets==1.10.0
azureml-pipeline==1.10.0
azureml-pipeline-core==1.10.0
azureml-pipeline-steps==1.10.0
azureml-sdk==1.10.0
azureml-telemetry==1.10.0
azureml-train==1.10.0
azureml-train-automl==1.10.0
azureml-train-automl-client==1.10.0
azureml-train-automl-runtime==1.10.0
azureml-train-core==1.10.0
azureml-train-restclients-hyperdrive==1.10.0
Babel==2.4.0
backcall==0.2.0
backports.shutil-get-terminal-size==1.0.0
backports.tempfile==1.0
backports.weakref==1.0.post1
beautifulsoup4==4.6.0
bitarray==0.8.1
blaze==0.10.1
bleach==1.5.0
bokeh==1.3.4
boto==2.49.0
boto3==1.9.247
botocore==1.12.247
Bottleneck==1.2.1
certifi==2019.9.11
cffi==1.12.3
chardet==3.0.4
chart-studio==1.0.0
click==6.7
cloudpickle==1.2.2
clyent==1.2.2
colorama==0.3.9
configparser==3.7.4
contextlib2==0.6.0.post1
cryptography==2.7
cycler==0.10.0
Cython==0.29.13
cytoolz==0.8.2
dask==0.14.3
datashape==0.5.4
decorator==4.4.0
dill==0.3.1.1
distributed==1.16.3
distro==1.4.0
docker==4.1.0
docutils==0.15.2
dotnetcore2==2.1.14
entrypoints==0.3
et-xmlfile==1.0.1
fastcache==1.0.2
fire==0.2.1
Flask==1.0.3
Flask-Cors==3.0.2
fsspec==0.5.2
fusepy==3.0.1
gast==0.3.2
gensim==3.8.1
gevent==1.2.1
google-pasta==0.1.7
greenlet==0.4.12
grpcio==1.24.1
gunicorn==19.9.0
h5py==2.10.0
HeapDict==1.0.1
html5lib==0.999
idna==2.8
imageio==2.6.1
imagesize==0.7.1
importlib-metadata==0.23
interpret-community==0.14.4
interpret-core==0.2.1
ipykernel==4.6.1
ipython==7.8.0
ipython-genutils==0.2.0
ipywidgets==6.0.0
isodate==0.6.0
isort==4.2.5
itsdangerous==0.24
jdcal==1.3
jedi==0.10.2
jeepney==0.4.1
Jinja2==2.10.3
jmespath==0.9.4
joblib==0.14.1
json-logging-py==0.2
JsonForm==0.0.2
jsonpickle==1.2
jsonschema==3.1.1
JsonSir==0.0.2
jupyter==1.0.0
jupyter-client==5.0.1
jupyter-console==5.1.0
jupyter-core==4.3.0
Keras-Applications==1.0.8
Keras-Preprocessing==1.1.0
keras2onnx==1.5.2
kiwisolver==1.1.0
lazy-object-proxy==1.2.2
liac-arff==2.5.0
library-metadata-cooker==0.0.1
lightgbm==2.2.3
llvmlite==0.18.0
locket==0.2.0
lxml==3.7.3
Markdown==3.1.1
MarkupSafe==1.1.1
matplotlib==3.1.1
missingno==0.4.2
mistune==0.7.4
mmlspark==1.0.0.dev1
more-itertools==7.2.0
mpmath==0.19
msal==1.5.0
msal-extensions==0.1.3
msgpack==0.6.2
msgpack-python==0.4.8
msrest==0.6.10
msrestazure==0.6.2
multimethods==1.0.0
multipledispatch==0.4.9
mypy==0.780
mypy-extensions==0.4.3
navigator-updater==0.1.0
nbconvert==5.1.1
nbformat==4.3.0
ndg-httpsclient==0.5.1
networkx==2.3
nimbusml==1.7.1
nltk==3.2.3
nose==1.3.7
notebook==5.0.0
notebookutils==2.4.4-20200918.3
numba==0.33.0
numexpr==2.6.2
numpy==1.16.2
numpydoc==0.6.0
oauthlib==3.1.0
odo==0.5.0
olefile==0.44
onnx==1.6.0
onnxconverter-common==1.6.0
onnxmltools==1.4.1
onnxruntime==1.0.0
openpyxl==2.4.7
packaging==19.2
pandas==0.23.4
pandocfilters==1.4.1
param==1.9.2
partd==1.0.0
pathlib2==2.2.1
pathspec==0.6.0
patsy==0.5.1
pep8==1.7.0
pexpect==4.2.1
pickleshare==0.7.4
Pillow==6.2.0
pip==9.0.1
plotly==4.1.1
ply==3.10
pmdarima==1.1.1
portalocker==1.7.1
prompt-toolkit==2.0.10
protobuf==3.10.0
psutil==5.2.2
ptyprocess==0.5.1
py==1.4.33
py-cpuinfo==5.0.0
py4j==0.10.7
pyarrow==0.15.1
pyasn1==0.4.7
pycosat==0.6.2
pycparser==2.19
pycrypto==2.6.1
pyct==0.4.6
pycurl==7.43.0
pyflakes==1.5.0
pygal==2.4.0
Pygments==2.2.0
PyJWT==1.7.1
pylint==1.6.4
pymssql==2.1.4
pyodbc==4.0.16
pyOpenSSL==19.0.0
pyparsing==2.4.2
pyrsistent==0.15.4
pyspark==2.4.4
pytest==3.0.7
python-dateutil==2.8.0
Python-EasyConfig==0.1.7
pytz==2019.3
PyWavelets==1.0.3
PyYAML==5.1.2
pyzmq==16.0.2
QtAwesome==0.4.4
qtconsole==4.3.0
QtPy==1.2.1
requests==2.22.0
requests-oauthlib==1.2.0
Resource==0.2.1
retrying==1.3.3
rope-py3k==0.9.4.post1
ruamel.yaml==0.16.10
ruamel.yaml.clib==0.2.2
s3transfer==0.2.1
scikit-image==0.15.0
scikit-learn==0.20.3
scipy==1.2.0
seaborn==0.9.0
SecretStorage==3.1.1
setuptools==41.4.0
shap==0.34.0
simplegeneric==0.8.1
singledispatch==3.4.0.3
six==1.12.0
skl2onnx==1.4.9
sklearn-pandas==1.7.0
smart-open==1.8.4
snowballstemmer==1.2.1
sortedcollections==0.5.3
sortedcontainers==2.1.0
Sphinx==1.5.6
spyder==3.1.4
SQLAlchemy==1.1.9
statsmodels==0.10.1
sympy==1.0
tables==3.3.0
tblib==1.4.0
tensorboard==1.14.0
tensorflow==1.14.0
tensorflow-estimator==1.14.0
termcolor==1.1.0
terminado==0.6
testpath==0.3
toolz==0.10.0
torch==1.4.0
tornado==6.0.3
tqdm==4.50.0
traitlets==4.3.2
typed-ast==1.4.1
typing-extensions==3.7.4
unicodecsv==0.14.1
urllib3==1.25.6
vega-datasets==0.7.0
wcwidth==0.1.7
websocket-client==0.56.0
Werkzeug==0.16.1
wheel==0.30.0
widgetsnbextension==2.0.0
wrapt==1.11.2
xgboost==0.90
xlrd==1.0.0
XlsxWriter==0.9.6
xlwt==1.2.0
zict==1.0.0
zipp==0.6.0
google-api-core==1.23.0
google-api-python-client==1.12.4
google-auth==1.22.1
google-auth-httplib2==0.0.4
google-auth-oauthlib==0.4.1
googleapis-common-protos==1.52.0
httplib2==0.18.1

azure-synapse-analytics
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

HimanshuSinha-MSFT avatar image
0 Votes"
HimanshuSinha-MSFT answered SanjeevNayak-3936 commented

Thanks @anadelcampo ,

As we connected offline and found that the filesystem under that the ADLS gen2 was mssing and adding that fixed the issue .

Thanks Himanshu
Please do consider to click on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members

· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Yes AND I am going to add that you need to load the requirements.txt file from the azure portal and NOT from the workspace:

35754-image.png


0 Votes 0 ·
image.png (46.4 KiB)

Hi, I am facing same issue, when uploading the requirements.txt file under poolname/packages folder looks like its not installing the libraries mantioned in the file.which filesystem was missing under ADLS gen2 ? could you please explain bit in details.

Regards
Sanjeev

0 Votes 0 ·

Hi HimanshuSinha-MSFT,

I am facing the same issue after uploading the requirements.txt file, the libraries are not getting installed, could you please mention the detailed step where to upload in ADLS Gen2 account
I am uploading at below path.
XXX/synapse/workspaces/northstarlabws1/sparkpools/sparkpool2

Your quick help would be appreciated.

Regards
Sanjeev

0 Votes 0 ·
HimanshuSinha-MSFT avatar image
0 Votes"
HimanshuSinha-MSFT answered HimanshuSinha-MSFT commented

Hello @anadelcampo,

Thanks for the ask and also using the Microsoft Q&A.
Since you never mentioned the library so I am notsure , but i do see the below installed as per your thread . Please scroll down till the end .

google-api-core==1.23.0
google-api-python-client==1.12.4
google-auth==1.22.1
google-auth-httplib2==0.0.4
google-auth-oauthlib==0.4.1
googleapis-common-protos==1.52.0*I

Thanks Himanshu
Please do consider to click on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members



· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

yes you are correct, those are the libraries that I have not been able to install. Do you mean you were able to see those libraries install in a cluster once the requirements.txt file was uploaded?

0 Votes 0 ·

Hello ,

I am able to install the libraries . Please check the snapshot below .

34731-capture.png



One more thing , the Synapse workspace is sattched to ADLS gen2 account and you need to have at least the blob contributor role access on that . The below snapsot should help you locate the account from the WS .

34741-capture2jpg.png
Please do let me know if you are still facing the issue .



0 Votes 0 ·
capture.png (28.7 KiB)
capture2jpg.png (20.9 KiB)
anadelcampo avatar image
0 Votes"
anadelcampo answered anadelcampo edited

That is weird because I had already granted those permissions to both myself and the synapse workspace, I tried it again for good measure but it did not work:

34791-image.png



image.png (25.4 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.