question

sunitasarma-7003 avatar image
0 Votes"
sunitasarma-7003 asked Pranay-9917 commented

Error when mounting Azure Data Lake Storage Gen1 account

I'm using the code below to mount gen1 account on azure databricks

configs = {"fs.adl.oauth2.access.token.provider.type": "ClientCredential",
"fs.adl.oauth2.client.id": <service-principal-appid>,
"fs.adl.oauth2.credential":<storage-account-access-key>,
"fs.adl.oauth2.refresh.url": "https://login.microsoftonline.com/<tenant-id>/oauth2/token"}
dbutils.fs.mount(
source = "adl://<storageaccountname>.azuredatalakestore.net/",
mount_point = "/mnt/<container>",
extra_configs = configs)
It mounts successfully, however, when I try to list the files using dbutils.fs.ls(/mnt/<container>), it throws the error below. The Service principal I'm using has owner role over the storage account.

An error occurred while calling z:com.databricks.backend.daemon.dbutils.FSUtils.ls.
: com.microsoft.azure.datalake.store.ADLException: Error enumerating directory /
Error fetching access token
Operation null failed with exception java.io.IOException : Server returned HTTP response code: 401 for URL:

azure-databricksazure-data-lake-storage
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Pranay-9917 avatar image
0 Votes"
Pranay-9917 answered

@sunitasarma-7003 If the hierarchical namespace is disabled then its not a DataLake, its a Blob Storage V2.
Could you try using blob storage mount:

dbutils.fs.mount(
source = "wasbs://<container-name>@<storage-account-name>.blob.core.windows.net",
mount_point = "/mnt/<mount-name>",
extra_configs = {"fs.azure.account.key.{0}.blob.core.windows.net".format(storage_account_name): storage_account_key}
)

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

SaurabhSharma-msft avatar image
0 Votes"
SaurabhSharma-msft answered SaurabhSharma-msft commented

@sunitasarma-7003 Thanks for using Microsoft Q&A !!

I believe you are not passing correct credentials while mounting ADLS account. You need to pass your service principal secret (client secret) to fs.adl.oauth2.credential and not <storage-account-access-key>. Can you please try again with the client secret and let me know if any issues.
You can get a client secret by going to Azure Portal > Azure Active Directory> App Registrations and select your registered application(service principal)

83418-image.png


Please do not forget to "Accept the answer" wherever the information provided helps you to help others in the community.




image.png (79.6 KiB)
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @sunitasarma-7003,

I have not heard back from you. Did my answer solve your issue? If so, please mark as accepted answer. If not, please let me know how I may better assist.

Thanks
Saurabh

0 Votes 0 ·
sunitasarma-7003 avatar image
0 Votes"
sunitasarma-7003 answered SaurabhSharma-msft commented

Hi @SaurabhSharma-msft Thank you for your response and apologies for the delay. I passed the client secret to fs.adl.oauth2.credential and I'm still receiving the same error. The service principal has Contributor role assigned to the storage account.

· 5
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@sunitasarma-7003 ok,

Can you please check if you are able to fetch the access token by using service to service authenticaton using the same client id/secret and use the ADLS Storage REST API using the returned token.


0 Votes 0 ·

@SaurabhSharma-msft I was able to retrieve the bearer token (service to service authentication). However I get the below error when trying to list folders using REST API
HTTPSConnectionPool(host='xxxxxxx.azuredatalakestore.net', port=443): Max retries exceeded with url: /webhdfs/v1/?op=LISTSTATUS (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x000002941AACF580>: Failed to establish a new connection:
[WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond'))

0 Votes 0 ·

@sunitasarma-7003 This means the host name is not correct or you do not have a route to reach to the data lake storage. Can you please check on the portal if the adls account name is correct ?
84154-image.png

Also, check if you are able to reach to your adls gen1 using nslookup xxxxxx.azuredatalakestore.net. Also, check under Firewall and virtual networks if Allow access is for all networks or selected networks ?
84164-image.png


0 Votes 0 ·
image.png (44.2 KiB)
image.png (41.7 KiB)

@sunitasarma-7003 Following up to know if you are still having issues ?

0 Votes 0 ·
Show more comments
sunitasarma-7003 avatar image
0 Votes"
sunitasarma-7003 answered KranthiPakala-MSFT converted comment to answer

Hi @SaurabhSharma-msft, I see that Allow access from all networks is checked. One quick question - under properties, I don't see the blob URL starting with adl:// . This is the blob service endpoint - https://xxxxxx.blob.core.windows.net/. And the account type is StorageV2 (general purpose v2). However, hierarchical namespace is disabled so I assumed it's gen1. Am I missing something here?
Appreciate your help!

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@sunitasarma-7003 If this is the case then you are not using a datalake it is a blob storage account and you need to mount blob storage account. Please refer to the documentation to do the same and update back how it goes.


0 Votes 0 ·
sunitasarma-7003 avatar image
0 Votes"
sunitasarma-7003 answered Pranay-9917 commented

@SaurabhSharma-msft , @Pranay-9917 I'm able to mount the blob storage successfully now :) sorry for the trouble and appreciate your help.

· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@sunitasarma-7003 glad to hear it worked. Could you please accept the answer.

Thanks

0 Votes 0 ·

I don't see an option to accept either of your replies (since it was a reply to my answer). Can the comments be moved to "write an answer" section.

0 Votes 0 ·

@sunitasarma-7003 yes. I just did.

Thanks

0 Votes 0 ·