Apache Spark in Azure Synapse Analytics - HTTP request in notebook
Hello.
I use a Notebook in Synapse where I run my Python code.
I would like to make an API request from this Notebook to Microsoft Purview to send the entities.
I added the pyapacheatlas library to spark.
On my local computer, this code works fine in Visual Studio.
I need to sign in with Microsoft. I created Purview client connections using a service principal. Here is the code that I am running:
from pyapacheatlas.auth import ServicePrincipalAuthentication
from pyapacheatlas.core import PurviewClient
from pyapacheatlas.core import AtlasEntity
auth = ServicePrincipalAuthentication(
tenant_id="...",
client_id="...",
client_secret="..."
)
# Create a client to connect to your service.
client = PurviewClient(
account_name = "...",
authentication = auth
)
# Get All Type Defs
all_type_defs = client.get_all_typedefs()
I am getting an error after running for a long time:
ConnectionError: HTTPSConnectionPool(host='login.microsoftonline.com', port=443): Max retries exceeded with url: /.../oauth2/token (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f00effc5250>: Failed to establish a new connection: [Errno 110] Connection timed out'))
It turned out that I can't make any HTTP requests in Notebook.
Even an ordinary GET like this:
import json
import requests
r = requests.get("http://echo.jsontest.com/insert-key-here/insert-value-here/key/value")
df = sqlContext.createDataFrame([json.loads(line) for line in r.iter_lines()])
As a result I get:
ConnectionError: HTTPConnectionPool(host='echo.jsontest.com', port=80): Max retries exceeded with url: /insert-key-here/insert-value-here/key/value (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f90b0111940>: Failed to establish a new connection: [Errno 110] Connection timed out'))
Please advise, maybe this is not provided by the functionality or is it possible to solve it?
Thank you.