When running the sample code "Explore NYC Yellow Taxi Data using Spark" in a PySpark notebook the Spark Pool starts successfully but the code just hangs and eventually returns with the following all though it has also returned with different errors.
I'm guessing this has something to do with the fact out storage account is locked down to specific Virtual Networks and Specific IP addresses. I have followed the steps to create a Managed Private Endpoint which points to the storage account and I can see the storage account in Data Browser on the left in Synapse Studio. I have also used the PowerShell command Add-AzStorageAccountNetworkRule to enable AAD pass through as per link: https://azureaggregator.wordpress.com/2021/01/08/storage-configuration-for-external-table-is-not-accessible-while-query-on-serverless-2/ but nothing has worked. Can anyone help on this pleaase?
"Resolving access token for scope "https://storage.azure.com/.default" using identity of type "USER".
Credentials are not provided to access data from the source. Please sign in using identity with required permission granted.
Interactive sign-in timeout: 120 sec.
To change the sign-in tenant, restart the session with tenant ID set to environment variable "AZUREML_DATA_ACCESS_TENANT_ID" before sign in.
To always use device code for interactive sign-in, set environment variable "AZUREML_DATA_ACCESS_USE_DEVICE_CODE" to "true".
To configure timeout, set environment variable "AZUREML_DATA_ACCESS_INTERACT_TIMEOUT" to the number of seconds."
Hello @David Leach ,
I am just checking in to see if you have any further questions here.