Azure HDInsight
An Azure managed cluster service for open-source analytics.
198 questions
1 answer
One of the answers was accepted by the question author.
Azure HD Insight Kafka Broker Setting for max.incremental.fetch.session.cache.slots
Hi, I need to change the setting for Broker Config for max.incremental.fetch.session.cache.slots . But I am not able to see this as a property on Ambari Broker Config. How can I change it ?
asked 2020-10-12T21:53:44.247+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(208, 4%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHA%3C/text%3E%3C/svg%3E)
Harjani, Ashish
201
Reputation points
accepted 2020-10-18T23:30:33.48+00:00
Harjani, Ashish
201
Reputation points
1 answer
Programmatic way to perform tasks on Azure HDinsight
Can someone please give reference to perform below tasks on Azure HDinsight via programmatic way: Enable diagnostics setting on HDinsights Cluster Enable Azure Monitor for the HDInsight cluster Enable NSG Flow Logs NSG Modifications (Rule Add,…
asked 2020-10-07T09:20:05.27+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(137.6, 61%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAV%3C/text%3E%3C/svg%3E)
Akash Verma
21
Reputation points
commented 2020-10-09T11:08:02.917+00:00
PRADEEPCHEEKATLA-MSFT
76,836
Reputation points Microsoft Employee
1 answer
How we link livy server is end point correct in HDinsight
Hi All, i want to submit my spark job from intellij to use adf spark activity. instead of head node host which IP address i have to give. here i give password of my admin account username/password but it is error out i am new…
asked 2020-09-23T14:56:17.643+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(262.40000000000003, 20%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMV%3C/text%3E%3C/svg%3E)
manish verma
421
Reputation points
commented 2020-10-07T13:06:32.897+00:00
PRADEEPCHEEKATLA-MSFT
76,836
Reputation points Microsoft Employee
1 answer
Error Setting Access Rights on ADL (Gen1) User Folders During HDInsight Cluster Creation
Hello All The issue occurs when the cluster is created. One of the last operations that Ambari performs are 'Post user creation hook for 1 users'. This fail with the attached errors. Here is some more information: It's an ESP cluster -- we…
asked 2020-09-22T07:21:55.527+00:00
Christoph Kiefer
141
Reputation points
commented 2020-10-07T13:04:47.837+00:00
PRADEEPCHEEKATLA-MSFT
76,836
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Spark unable to write file onto Blob storage
We use HDInsight with Spark, v3.6. So far, our code has been working as expected. As of last night, our job started failing. The error states that "output directory already exists". When looking at the blob storage, directories appear to be…
asked 2020-10-02T16:25:54.1+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(73.60000000000001, 65%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESS%3C/text%3E%3C/svg%3E)
Sachin Shah
101
Reputation points
accepted 2020-10-07T12:05:08.527+00:00
Sachin Shah
101
Reputation points
0 answers
Is it possible to send data from WPF app to Azure analytics?
Is it possible to send data from WPF app to Azure analytics? If yes, which is the minimum Visualstudio version & .net framework version required ? which are the best analytic services offered by Azure which can used with WPF app?
asked 2020-10-01T09:42:53.153+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(32, 71%, 13%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EH%3C/text%3E%3C/svg%3E)
Hi
1
Reputation point
commented 2020-10-07T00:46:17.98+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(172.8, 70%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
MartinJaffer-MSFT
26,021
Reputation points
2 answers
One of the answers was accepted by the question author.
How i call a Hdinsight Zeppelin notebook in Azure Data Factory
How i call a Hdinsight Zeppelin notebook in Azure Data Factory. i have a Zeppelin notebook in Hdinsight cluster , i need to call in azure data factory. what is best way to write spark code , and call in data factory.
asked 2020-09-21T11:31:03.803+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(153.6, 57.00000000000001%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVM%3C/text%3E%3C/svg%3E)
Verma, Manish Kumar
131
Reputation points
accepted 2020-10-02T08:22:44.707+00:00
Verma, Manish Kumar
131
Reputation points
1 answer
How to add ambari infra to a hbase cluster?
I tried to add it by add service link next to it in the admin section but add service link is not responding. Is there any other way to add this service?
asked 2020-09-17T05:52:27.67+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(236.8, 50%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAB%3C/text%3E%3C/svg%3E)
Ayushi Bhandari
1
Reputation point
commented 2020-09-30T12:16:45.453+00:00
PRADEEPCHEEKATLA-MSFT
76,836
Reputation points Microsoft Employee
1 answer
Apache Atlas on azure hdinsigt
How to install apache atlas on azure HDinsight?
asked 2020-09-16T08:24:28.39+00:00
Ayushi Bhandari
1
Reputation point
commented 2020-09-30T12:16:25.453+00:00
PRADEEPCHEEKATLA-MSFT
76,836
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Data Analytics Studio (DAS) on HDInsight Cluster
Dear All I am following the recommendations here to install the Data Analytics Studio (from Cloudera): https://learn.microsoft.com/en-us/azure/hdinsight/interactive-query/apache-hive-migrate-workloads The custom script action is not working…
asked 2020-09-22T11:56:27.23+00:00
Christoph Kiefer
141
Reputation points
accepted 2020-09-30T07:11:23.807+00:00
Christoph Kiefer
141
Reputation points
1 answer
One of the answers was accepted by the question author.
remainingCores is too small error. Spark and Kafka cluster on HDInsight problem
I tried to create a Kafka and Spark cluster in the HDInsight using either of templates in these links: …
asked 2020-09-27T06:57:32.923+00:00
Erfan Mahmoodnejad
41
Reputation points
commented 2020-09-29T06:31:58.623+00:00
PRADEEPCHEEKATLA-MSFT
76,836
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Create On Demand HDInsight from Data Factory in an existing VNet
The Azure Data Factory can create an on-demand HDInsight (linked service), which is automatically deleted after the job running on it finishes. Is it possible to have this on-demand HDInsight created in an existing VNet?
asked 2020-09-23T07:07:55.617+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(140.8, 4%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBM%3C/text%3E%3C/svg%3E)
Beniamin Murza
26
Reputation points
commented 2020-09-28T12:11:44.993+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(108.80000000000001, 9%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HarithaMaddi-MSFT
10,126
Reputation points
1 answer
Apache Kafka as a Service with Confluent Cloud Azure
We are trying to connect to on premise server from our Apache Kafka as a Service with Confluent Cloud; with on-premise SQL Server source, but receive below error Do we require any specials setting from azure cloud to on-premise?
asked 2020-09-26T04:42:30.917+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(198.4, 48%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDN%3C/text%3E%3C/svg%3E)
Dondapati, Navin
281
Reputation points
answered 2020-09-28T09:59:03.723+00:00
PRADEEPCHEEKATLA-MSFT
76,836
Reputation points Microsoft Employee
0 answers
creating conda environment on azure hdinsight spark cluster taking hours
trying to follow the code in this tutorial , https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/training/train-in-spark/train-in-spark.ipynb when i send the run to run to azure ml, the run essentially stalls on the step…
asked 2020-09-10T14:14:09.1+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(96, 1%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKK%3C/text%3E%3C/svg%3E)
kunal kapoor
1
Reputation point
commented 2020-09-15T22:43:58.83+00:00
MartinJaffer-MSFT
26,021
Reputation points
0 answers
Linking Blob storage with azure-hdinsight table
Hi All, I am new to Azure data lake. My requirement is like I need to store image and each image refers to advertisement (could be of string type). For this I have stored images in azure storage account and advertisement is stored in azure-hdinsight…
asked 2020-08-19T06:34:03.86+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(262.40000000000003, 86%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAV%3C/text%3E%3C/svg%3E)
Archana Vaidya
1
Reputation point
commented 2020-09-10T19:58:25.96+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 26%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HimanshuSinha-msft
19,376
Reputation points Microsoft Employee
1 answer
Security Recommendations for Azure Data and Analytics Services
I am working on Securing Data and Analytics Services on Azure. I want to know what security controls i can apply after creating of services and what i can apply only during the service creation. Below are the recommendation i have found as of now. Could…
asked 2020-09-02T20:20:43.98+00:00
Akash Verma
21
Reputation points
commented 2020-09-09T11:11:32.767+00:00
PRADEEPCHEEKATLA-MSFT
76,836
Reputation points Microsoft Employee
1 answer
Link HDInsight Cluster in VSCode to Show Hive Tables
Dear All I am using the latest release of VSCode 1.48.2 to connect to my HDInsight Spark Cluster (HDI 3.6, Spark 2.3). It successfully lists the Hive Databases available in my cluster when I browse to the 'Hive Databases' section insight the…
asked 2020-08-31T09:12:53.597+00:00
Christoph Kiefer
141
Reputation points
commented 2020-09-08T04:52:52.977+00:00
PRADEEPCHEEKATLA-MSFT
76,836
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
HDInsight Spark Cluster Customization with Boostrapping and Custom Action Scripts
Hello All We use both bootstrapping (via ARM templates) and action scripts to provision our HDInsight Spark Cluster (HDI 3.6, Spark 2.3). We face several challenges (in no particular order): First, some of the bootstrapping statements are not…
asked 2020-08-27T18:29:32.387+00:00
Christoph Kiefer
141
Reputation points
commented 2020-09-03T05:35:49.9+00:00
PRADEEPCHEEKATLA-MSFT
76,836
Reputation points Microsoft Employee
2 answers
Networking Issue on Azure HDInsight Spark Cluster with ESP
Dear All We encounter an issue with networking / DNS on our Azure HDInsight Spark cluster. The cluster is joined to our AAD (i.e., it's a cluster with ESP enabled). The cluster gets automatically created with a PS runbook and ARM template file. This…
asked 2020-08-24T09:42:23.157+00:00
Christoph Kiefer
141
Reputation points
commented 2020-09-01T11:05:27.24+00:00
PRADEEPCHEEKATLA-MSFT
76,836
Reputation points Microsoft Employee
0 answers
Configure HDFS Storage for Zeppelin Notebooks on HDInsight Spark Clusters with ESP
Dear All We followed this step-by-step tutorial to configure HDFS storage for Zeppelin notebooks on our ESP-enabled HDInsight Spark Cluster (HDI 3.9, Spark 2.3):…
asked 2020-08-28T09:55:45.47+00:00
Christoph Kiefer
141
Reputation points
commented 2020-09-01T08:18:44.543+00:00
PRADEEPCHEEKATLA-MSFT
76,836
Reputation points Microsoft Employee