Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,338 questions
How to execute Hive query in Databricks?
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(195.2, 34%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESD%3C/text%3E%3C/svg%3E)
Suman Dutta
1
Reputation point
We are calling a ".jar" file from Azure Data Factory using Databricks JAR activity. In the JAR activity we are specifying the Cluster Id in Databricks Linked Service.
In Databricks cluster we are adding below Spark config:
spark.hadoop.hive.warehouse.subdir.inherit.perms true
spark.databricks.delta.preview.enabled true
spark.hadoop.dfs.adls.adls2name.hostname adls2name.dfs.core.windows.net
spark.sql.warehouse.dir abfss://clusters@adls2name.blob.core.windows.net/nonprod/warehouse/
spark.hadoop.dfs.adls.adls2name.mountpoint /clusters/nonprod/
spark.hadoop.dfs.adls.home.hostname adls2name.dfs.core.windows.net
spark.hadoop.dfs.adls.home.mountpoint /clusters/foldername/
spark.hadoop.dfs.adls.oauth2.refresh.url https://login.microsoftonline.com/xxxxxxxxxx/oauth2/token
spark.hadoop.dfs.adls.oauth2.client.id
spark.hadoop.dfs.adls.oauth2.credential
spark.hadoop.dfs.adls.oauth2.access.token.provider.type ClientCredential
spark.hadoop.fs.azure.account.key.stagedata01.blob.core.windows.net
spark.hadoop.fs.azure.account.key.adls2name.blob.core.windows.net
The jar file is in "stagedata01" adls. The jar file is copying file from one location to other location in "adls2name" adls.
But it is failing...
Please help how to resolve?
{count} votes
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 26%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HimanshuSinha-msft 19,376 Reputation points Microsoft Employee2022-02-14T21:50:11.957+00:00 Hello @Suman Dutta ,
Thanks for the ask and using Microsoft Q&A platform .
You mentioned that you are getting an error , can you please more details on the same please ? Also what is the logic in the JAR .
Are you able to look into the logs on the ADb cluster and see if any error is logged there also ?
Waiting to hear back from you on this .
Thanks
Himanshu -
HimanshuSinha-msft 19,376 Reputation points Microsoft Employee
2022-02-16T19:47:50.707+00:00 Hello @Suman Dutta ,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .In case if you have any resolution please do share that same with the community as it can be helpful to others . Otherwise, will respond back with the more details and we will try to help .
Thanks
Himanshu
Sign in to comment