Azure HDInsight
An Azure managed cluster service for open-source analytics.
199 questions
0 answers
Azure login audit logs not accurate
I have a user on Azure HD that is showing one failed login on 2/17/22 when doing a search in the Sign-in logs for a time frame on 1 month. I know for a fact that this user successfully logged in several times on and after 1/24/22 but nothing shows except…
asked 2022-02-18T18:47:10.96+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(105.60000000000001, 27%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERF%3C/text%3E%3C/svg%3E)
Ronald Flourish
1
Reputation point
commented 2022-02-22T13:23:22.677+00:00
ShaikMaheer-MSFT
38,201
Reputation points Microsoft Employee
0 answers
How to execute Hive query in Databricks?
We are calling a ".jar" file from Azure Data Factory using Databricks JAR activity. In the JAR activity we are specifying the Cluster Id in Databricks Linked Service. In Databricks cluster we are adding below Spark config: …
asked 2022-02-11T12:21:07.377+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(195.2, 34%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESD%3C/text%3E%3C/svg%3E)
Suman Dutta
1
Reputation point
commented 2022-02-16T19:47:50.707+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 26%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HimanshuSinha-msft
19,381
Reputation points Microsoft Employee
1 answer
Azure Data Residency and GDPR Compliance Criteria
All Our Current Azure Resources , Workloads and SQL Databases are located in the West-Europe region. Now we want to create new SQL Databases in the US East, US West regions and want to use re use existing Azure workloads, Just checking is it violates…
asked 2022-02-11T10:54:43.41+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(83.2, 9%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ER%3C/text%3E%3C/svg%3E)
RKG
1
Reputation point
commented 2022-02-15T17:16:30.12+00:00
ShaikMaheer-MSFT
38,201
Reputation points Microsoft Employee
1 answer
Active worker nodes usage in HDinsight cluster for the period of last one year
Hi Team, Can someone help us to get the Active worker nodes usage in HDinsight cluster for the period of last one year. Thanks & Regards, Udaya
asked 2022-01-24T12:50:17.057+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(259.20000000000005, 70%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EUS%3C/text%3E%3C/svg%3E)
UDAYA SRINIVASARAO KOTHAMASU
1
Reputation point
commented 2022-02-03T19:48:42.66+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(172.8, 70%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
MartinJaffer-MSFT
26,036
Reputation points
1 answer
HDinsight spark livy server stopping as soon as It starts
I am a new user of Azure HDInsight. I have created a 4 node spark cluster. When the cluster is successfully created, I saw spark history server and livy both are in a stopped state. And when I try to run them from Ambri it gets stopped by…
asked 2022-01-16T18:37:09.537+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(214.4, 37%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGS%3C/text%3E%3C/svg%3E)
Gourav Sharma
1
Reputation point
commented 2022-01-24T20:23:37.683+00:00
HimanshuSinha-msft
19,381
Reputation points Microsoft Employee
1 answer
HDInsight Kafka Disaster Recovery Solution
Hi , We need to create a DR strategy for HDInsight Kafka Cluster . We thought of following cross region unidirectional replication via mirror maker from primary to secondary cluster. The questions however We have are- -What kind of RTO and RPO we…
asked 2022-01-07T13:10:37.33+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(54.400000000000006, 74%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECM%3C/text%3E%3C/svg%3E)
Chandra Manral
1
Reputation point
commented 2022-01-24T16:49:25.893+00:00
ShaikMaheer-MSFT
38,201
Reputation points Microsoft Employee
1 answer
Accessing HBase on HDInsight cluster via public internet
Hi, Is there a way to access HBase cluster on HDInsight from public internet, not within Azure infrastructure? I'm trying to migrate smoothly from GCP bigtable to HBase on Azure HDInsight and is needed to access Azure's HBase cluster from our current…
asked 2022-01-06T09:13:37.287+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(272, 45%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EW%3C/text%3E%3C/svg%3E)
wooseok
1
Reputation point
commented 2022-01-13T10:29:12.287+00:00
PRADEEPCHEEKATLA-MSFT
79,551
Reputation points Microsoft Employee
4 answers
One of the answers was accepted by the question author.
Upgrade HDInsight Python Version
I know that HDInsight is using Python 3.5, but is there a way to upgrade the minor version to Python 3.6 or above? The reason is that we have a third party package which only works on Python 3.6 or above. …
asked 2020-10-26T12:34:49.347+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(185.6, 22%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EH%3C/text%3E%3C/svg%3E)
HenryX
21
Reputation points
commented 2021-12-23T07:26:55.54+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(259.20000000000005, 19%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESA%3C/text%3E%3C/svg%3E)
Sarthak Agrawal
1
Reputation point Microsoft Employee
1 answer
Log4J vulnerability azure HDinsights
As there is a Log4J vulnerability trending recently. May I get clarifications for the below points. 1) How the Log4J vulnerability impacting HDInsight service ? Any Impact on Yarn/Hive/Spark logging utilities 2) How can I prevent or take precautions from…
asked 2021-12-14T06:38:47.437+00:00
UDAYA SRINIVASARAO KOTHAMASU
1
Reputation point
commented 2021-12-22T12:05:03.9+00:00
PRADEEPCHEEKATLA-MSFT
79,551
Reputation points Microsoft Employee
0 answers
AmbariClusterCreationFailedErrorCode
Unable to launch an HDInsight ESP enabled cluster. Getting below error: { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations…
asked 2021-12-12T15:41:59.85+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(92.8, 10%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESA%3C/text%3E%3C/svg%3E)
Shamsher Ansari
1
Reputation point
commented 2021-12-13T07:25:39.723+00:00
PRADEEPCHEEKATLA-MSFT
79,551
Reputation points Microsoft Employee
1 answer
Connect HdInsight to Local Superset
Hello, I have an HDInsight cluster and Superset installed in my local machine. So, i want to know if is possible to conect Superset to my HdInsight Cluster. Best Regards, Paulo
asked 2021-11-18T11:59:43.217+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(108.80000000000001, 24%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPB%3C/text%3E%3C/svg%3E)
Paulo Barbosa
21
Reputation points
commented 2021-11-29T18:22:44.137+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 9%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKM%3C/text%3E%3C/svg%3E)
KranthiPakala-MSFT
46,427
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
How to access to HDFS namenode UI
Hello, I have an HdInight cluster, but I can't access to the HDFS namenode UI. In Ambari, the link to access HDFS namenode UI is: https://{CLUSTERNAME}/da/host/hn0-testlo.nmfkjwercu3efldgcwsgpmaqge.ax.internal.cloudapp.net/port/30070/ But, when I…
asked 2021-11-18T12:08:25.937+00:00
Paulo Barbosa
21
Reputation points
accepted 2021-11-22T16:18:05.23+00:00
Paulo Barbosa
21
Reputation points
1 answer
azure shared dashboard contents deleted.
Hello All, We have a shared dashboard in all environments and we have lost all contents in dev and PR. The dashboard is still present and only the content is lost and we get a empty screen without any tiles or blocks, just empty page. We did not…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(22.400000000000002, 19%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDB%3C/text%3E%3C/svg%3E)
1 answer
One of the answers was accepted by the question author.
Azure HD insight cluster - Patching & Upgrading
We have a Kakfa cluster (Azure HD Insights) running on Azure. Does this need a manual intervention for OS Patching & Updates? https://learn.microsoft.com/en-us/azure/hdinsight/hdinsight-os-patching Is it not handled by Azure?
asked 2021-11-12T13:56:44.403+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(275.2, 43%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EST%3C/text%3E%3C/svg%3E)
Senthilnath TM
241
Reputation points
accepted 2021-11-18T13:08:21.39+00:00
Senthilnath TM
241
Reputation points
1 answer
Create Hive database with Jupyter Notebook
I have a HDInsight cluster and I want to create hive databases and tables (and load data into them) using Jupyter Notebook. Can anyone explain how can I do that? Is there any type of example notebooks explaining that?
asked 2021-10-25T17:08:54.737+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(169.60000000000002, 51%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDR%3C/text%3E%3C/svg%3E)
Diogo Rodrigues
1
Reputation point
commented 2021-11-02T03:27:18.38+00:00
PRADEEPCHEEKATLA-MSFT
79,551
Reputation points Microsoft Employee
0 answers
HdInsight problem when create database in Hive from Jupyter Notebook
I want to create a new database in hive, read and write data. So, i created a new jupyter notebook file with spark kernel and followed a tutorial. At the end, i can see the database and the table created in jupyter but i can´t see the database in Hive…
asked 2021-10-28T09:27:03.357+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(272, 92%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPB%3C/text%3E%3C/svg%3E)
Paulo Barbosa
1
Reputation point
commented 2021-11-02T03:26:46.95+00:00
PRADEEPCHEEKATLA-MSFT
79,551
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
How can i sync azure HDinsight data to another hdinsight in different region?
We are planning to migrate hdinsight from East Asia to some region else. Is there any tool we can use to do the migration automatically?
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(233.6, 61%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
0 answers
Reverse lookup between two HDI cluster fails
Reverse lookup between two clusters fail, we have created script action to update the host files on our spark worker nodes. When scaling up the new worker nodes in spark have appropriate host entries to connect to hbase but when spark is scaled down…
asked 2021-10-22T00:34:58.117+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(64, 59%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVS%3C/text%3E%3C/svg%3E)
Varun Sharma
81
Reputation points
commented 2021-10-27T06:09:36.82+00:00
PRADEEPCHEEKATLA-MSFT
79,551
Reputation points Microsoft Employee
1 answer
Difference between synapse and hdinsight
What are major difference between synapse and HD insight and under what circumstances one should use synapse and vice versa
asked 2021-10-12T12:43:25.457+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(281.6, 34%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESR%3C/text%3E%3C/svg%3E)
Shivam Ramola
66
Reputation points
commented 2021-10-19T15:37:11.57+00:00
MartinJaffer-MSFT
26,036
Reputation points
1 answer
Accessing adls gen2 from an esp enabled HDinsights cluster
Hallo All, I have configured an HDI Cluster with ESP enabled and from the Ambari and Azure Console I can see that all the services are running fine. But when trying to execute a sample pi job even to list files using hdfs dfs -ls , I am getting the…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(54.400000000000006, 26%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVB%3C/text%3E%3C/svg%3E)