Azure HDInsight
An Azure managed cluster service for open-source analytics.
207 questions
1 answer
Can't create HDInsight Cluster(Hadoop)
Dear All, I am struggling against creating HDInsight. After reviewing documents and other posts, I upgraded free-trial to paid subscription and created paid subscription as well. However, regardless of subscription types(both subscription paid type) I…
asked 2022-05-03T15:57:25.88+00:00
Jongmin Lee
6
Reputation points
commented 2022-05-09T05:15:42.647+00:00
PRADEEPCHEEKATLA-MSFT
85,826
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
How to leverage Azure key vault secrets from HD Insight Jupyter notebook?
Hi, I am trying to store the user id and password in Secrets and retrieve them in HD Insight Jupyter notebook? Any guidance.
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(48, 91%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJ%3C/text%3E%3C/svg%3E)
1 answer
One of the answers was accepted by the question author.
How to use GnuPG in HDInsight for encryption and decryption?
Hi, I am working with the HDInsight Spark cluster on Azure. Trying to encrypt files with pgp encryption using our private key. Is there a way that this can achieve rather than using the inbuilt encryption mechanism? How to set the home for GnuPG…
asked 2022-04-17T03:02:56.34+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(249.60000000000002, 81%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVS%3C/text%3E%3C/svg%3E)
vijay singh parmar
26
Reputation points
accepted 2022-04-28T07:21:51.667+00:00
vijay singh parmar
26
Reputation points
1 answer
Not enough cores error while deploying resource group on Azure for Students
Hello! I am very new to Azure. I have an Azure for Students subscription and I'm trying to create an Apache Kafka cluster using Azure HDInsight. I selected West Europe as my region. I'm using this resource as a guide:…
asked 2022-04-15T00:28:01.03+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(86.4, 32%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ELM%3C/text%3E%3C/svg%3E)
Leila Moussa
1
Reputation point
commented 2022-04-22T03:57:47.553+00:00
PRADEEPCHEEKATLA-MSFT
85,826
Reputation points Microsoft Employee
1 answer
Zeppelin notebook - sc.textFile does not work for HDI with ESP
We have HDI cluster with ESP enabled. From our zeppelin notebook, when I read data to a dataset (spark.read.text) it works but when I try to read it to an RDD (sc.textFile), I get an authentication exception: Note that, while sc.textFile…
asked 2021-02-11T15:14:14.283+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(44.800000000000004, 71%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESL%3C/text%3E%3C/svg%3E)
Steven Lai
1
Reputation point
commented 2022-04-01T04:03:14.057+00:00
PRADEEPCHEEKATLA-MSFT
85,826
Reputation points Microsoft Employee
0 answers
unable to access index for repository https://mran.microsoft.com/snapshot/2017-03-15/src/contrib
The last time I did same thing last month, it was still ok, but today When I tried to install R package from MRAN repository; I got this error Checking the repository via browser also error could not find repository Could you please help me in the…
asked 2022-03-19T10:46:16.717+00:00
Suryanto
16
Reputation points
commented 2022-03-21T22:31:04.06+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 44%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESS%3C/text%3E%3C/svg%3E)
Saurabh Sharma
23,791
Reputation points Microsoft Employee
1 answer
HDInsight: Commands to clean up the space
Hi, at my workplace, we are using HDInsight 3.6. We have encountered space issues before, but we were able to resolve them by simply executing the simple cleanup commands from the edge node. Unfortunately, these commands have not been useful recently.…
asked 2022-03-10T19:51:28.77+00:00
vijay singh parmar
26
Reputation points
commented 2022-03-17T10:07:15.783+00:00
PRADEEPCHEEKATLA-MSFT
85,826
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
run job in HDInsight compute linked service
is username and password is the only way to submit job to HDInsight cluster? is managed identity or msi or service principal supported? Added question: can HDI team build API which uses AAD tokens as password instead of user input password? we have…
asked 2022-03-04T21:52:05.777+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(0, 79%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBK%3C/text%3E%3C/svg%3E)
Bill Kan
21
Reputation points
commented 2022-03-11T03:03:01.097+00:00
PRADEEPCHEEKATLA-MSFT
85,826
Reputation points Microsoft Employee
0 answers
Run C# mapreduce job
I am a beginner to Hadoop MapReduce. I have implemented a MapReduce job in visual C# and want to run it locally. As I understood, the HDInsight emulator hasn't been updated for a long time. What else options I have, to run the job locally?
asked 2022-02-17T17:59:18.127+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(89.60000000000001, 86%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ELS%3C/text%3E%3C/svg%3E)
Lilukshi Silva
1
Reputation point
commented 2022-02-25T20:30:24.82+00:00
Lilukshi Silva
1
Reputation point
0 answers
Azure login audit logs not accurate
I have a user on Azure HD that is showing one failed login on 2/17/22 when doing a search in the Sign-in logs for a time frame on 1 month. I know for a fact that this user successfully logged in several times on and after 1/24/22 but nothing shows except…
asked 2022-02-18T18:47:10.96+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(105.60000000000001, 27%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERF%3C/text%3E%3C/svg%3E)
Ronald Flourish
1
Reputation point
commented 2022-02-22T13:23:22.677+00:00
ShaikMaheer-MSFT
38,416
Reputation points Microsoft Employee
0 answers
How to execute Hive query in Databricks?
We are calling a ".jar" file from Azure Data Factory using Databricks JAR activity. In the JAR activity we are specifying the Cluster Id in Databricks Linked Service. In Databricks cluster we are adding below Spark config: …
asked 2022-02-11T12:21:07.377+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(195.2, 34%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESD%3C/text%3E%3C/svg%3E)
Suman Dutta
1
Reputation point
commented 2022-02-16T19:47:50.707+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 26%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HimanshuSinha-msft
19,386
Reputation points Microsoft Employee
1 answer
Azure Data Residency and GDPR Compliance Criteria
All Our Current Azure Resources , Workloads and SQL Databases are located in the West-Europe region. Now we want to create new SQL Databases in the US East, US West regions and want to use re use existing Azure workloads, Just checking is it violates…
asked 2022-02-11T10:54:43.41+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(83.2, 9%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ER%3C/text%3E%3C/svg%3E)
RKG
1
Reputation point
commented 2022-02-15T17:16:30.12+00:00
ShaikMaheer-MSFT
38,416
Reputation points Microsoft Employee
1 answer
HDinsight spark livy server stopping as soon as It starts
I am a new user of Azure HDInsight. I have created a 4 node spark cluster. When the cluster is successfully created, I saw spark history server and livy both are in a stopped state. And when I try to run them from Ambri it gets stopped by…
asked 2022-01-16T18:37:09.537+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(214.4, 37%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGS%3C/text%3E%3C/svg%3E)
Gourav Sharma
1
Reputation point
commented 2022-01-24T20:23:37.683+00:00
HimanshuSinha-msft
19,386
Reputation points Microsoft Employee
1 answer
HDInsight Kafka Disaster Recovery Solution
Hi , We need to create a DR strategy for HDInsight Kafka Cluster . We thought of following cross region unidirectional replication via mirror maker from primary to secondary cluster. The questions however We have are- -What kind of RTO and RPO we…
asked 2022-01-07T13:10:37.33+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(54.400000000000006, 74%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECM%3C/text%3E%3C/svg%3E)
Chandra Manral
1
Reputation point
commented 2022-01-24T16:49:25.893+00:00
ShaikMaheer-MSFT
38,416
Reputation points Microsoft Employee
1 answer
Accessing HBase on HDInsight cluster via public internet
Hi, Is there a way to access HBase cluster on HDInsight from public internet, not within Azure infrastructure? I'm trying to migrate smoothly from GCP bigtable to HBase on Azure HDInsight and is needed to access Azure's HBase cluster from our current…
asked 2022-01-06T09:13:37.287+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(272, 45%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EW%3C/text%3E%3C/svg%3E)
wooseok
1
Reputation point
commented 2022-01-13T10:29:12.287+00:00
PRADEEPCHEEKATLA-MSFT
85,826
Reputation points Microsoft Employee
4 answers
One of the answers was accepted by the question author.
Upgrade HDInsight Python Version
I know that HDInsight is using Python 3.5, but is there a way to upgrade the minor version to Python 3.6 or above? The reason is that we have a third party package which only works on Python 3.6 or above. …
asked 2020-10-26T12:34:49.347+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(185.6, 22%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EH%3C/text%3E%3C/svg%3E)
HenryX
21
Reputation points
commented 2021-12-23T07:26:55.54+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(259.20000000000005, 19%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESA%3C/text%3E%3C/svg%3E)
Sarthak Agrawal
1
Reputation point Microsoft Employee
1 answer
Log4J vulnerability azure HDinsights
As there is a Log4J vulnerability trending recently. May I get clarifications for the below points. 1) How the Log4J vulnerability impacting HDInsight service ? Any Impact on Yarn/Hive/Spark logging utilities 2) How can I prevent or take precautions from…
asked 2021-12-14T06:38:47.437+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(259.20000000000005, 70%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EUS%3C/text%3E%3C/svg%3E)
UDAYA SRINIVASARAO KOTHAMASU
1
Reputation point
commented 2021-12-22T12:05:03.9+00:00
PRADEEPCHEEKATLA-MSFT
85,826
Reputation points Microsoft Employee
0 answers
AmbariClusterCreationFailedErrorCode
Unable to launch an HDInsight ESP enabled cluster. Getting below error: { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations…
asked 2021-12-12T15:41:59.85+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(92.8, 10%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESA%3C/text%3E%3C/svg%3E)
Shamsher Ansari
1
Reputation point
commented 2021-12-13T07:25:39.723+00:00
PRADEEPCHEEKATLA-MSFT
85,826
Reputation points Microsoft Employee
1 answer
Connect HdInsight to Local Superset
Hello, I have an HDInsight cluster and Superset installed in my local machine. So, i want to know if is possible to conect Superset to my HdInsight Cluster. Best Regards, Paulo
asked 2021-11-18T11:59:43.217+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(108.80000000000001, 24%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPB%3C/text%3E%3C/svg%3E)
Paulo Barbosa
21
Reputation points
commented 2021-11-29T18:22:44.137+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 9%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKM%3C/text%3E%3C/svg%3E)
KranthiPakala-MSFT
46,447
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
How to access to HDFS namenode UI
Hello, I have an HdInight cluster, but I can't access to the HDFS namenode UI. In Ambari, the link to access HDFS namenode UI is: https://{CLUSTERNAME}/da/host/hn0-testlo.nmfkjwercu3efldgcwsgpmaqge.ax.internal.cloudapp.net/port/30070/ But, when I…
asked 2021-11-18T12:08:25.937+00:00
Paulo Barbosa
21
Reputation points
accepted 2021-11-22T16:18:05.23+00:00
Paulo Barbosa
21
Reputation points