Synapse Workspace Notebooks - Upload files to sparkpools for google api credentials

Question

I am working with Synapse Workspace Notebooks to import information from google API's. I need to upload credential files to the local directories on the synapse spark pools, I have not been able to figure out how to do that

Uploading to azure storage and pointing the files from there won't when using google python provided libraries look at the local computer storage.

Answer 1

I am working with py.drive, I found a good work around that will do for now. In the notebook, I created a variable with the credentials then saved the credentials to the nodes. But I had no problem doing this on databricks, but synapse notebooks not the best way to do it.

CREDENTIALS = '{"web":{"client_id":"xxxxxxx.....}'

SECRETS = '{"installed":{"client_id":"xxxxxxxx.....}}'

MYCREDS = '{"access_token": "xxxxxxxx....."_module": "oauth2client.client"}'

import json
creds = json.loads(CREDENTIALS)
with open('credentials.json', 'w') as outfile:
json.dump(creds, outfile)

secrets = json.loads(SECRETS)
with open('client_secrets.json', 'w') as outfile:
json.dump(secrets, outfile)

mycreds = json.loads(MYCREDS)
with open('mycreds.txt', 'w') as outfile:
json.dump(mycreds, outfile)

from there I was able to continue with authentication code and so on.

Answer 2

You can use the normal python file APIs to interact with the actual filesystem on the node running the python code (the driver usually, but could be a worker if you are using a python UDF).

You may not be able to write a file to the location you need due to file permissions.