Azure HDInsight
An Azure managed cluster service for open-source analytics.
197 questions
2 answers
One of the answers was accepted by the question author.
Cores quota limit in HDInsight
I am trying to create HDInsight Spark cluster, however getting below error : You have reached your subscription's cores quota limit in France Central. Please choose a different region or request billing support to increase your limit for France…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(83.2, 24%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EV%3C/text%3E%3C/svg%3E)
1 answer
One of the answers was accepted by the question author.
ConnectException org.fluentd.logger.sender.RawSocketSender with Azure Monitor Integration
When we enabled Azure Monitor on our HDInsight cluster (https://learn.microsoft.com/en-us/azure/hdinsight/log-analytics-migration) we got the following exception when running Spark jobs: 22/04/06 12:15:29 ERROR RawSocketSender…
asked 2022-04-06T13:39:00.22+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(182.40000000000003, 70%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ETS%3C/text%3E%3C/svg%3E)
Thomas Schouten
61
Reputation points
commented 2024-03-20T07:03:52.61+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(310.4, 60%, 40%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVM%3C/text%3E%3C/svg%3E)
Venkataswamy M
25
Reputation points
1 answer
How to run spark streaming on azure synapse (which activity to choose to have a continuous run)
Hello community! I'm looking for the best way to run spark streaming on synapse (data is coming from event hub). I was thinking about Spark job definition, but spark job definition has a limited amount of time an activity can run (Default is 12 hours,…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(243.2, 23%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EM%3C/text%3E%3C/svg%3E)
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 61%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EP%3C/text%3E%3C/svg%3E)
1 answer
One of the answers was accepted by the question author.
HDInsight in Trial account.
I resister for HDInsight separately in my Trial account. Can I create HDInsight Clusters under Free Trial subscription or will there be any cost implications?
1 answer
InvalidNetworkConfigurationErrorCode Error message Virtual Network configuration is incompatible with HDInsight requirements. Error: 'Cluster Validation timed out.',
created hdinsight cluster for kafka but it is failing . Went through the documentation but unable to resolve
asked 2024-02-20T19:04:35.56+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(67.2, 12%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJS%3C/text%3E%3C/svg%3E)
Jitesh Shivnani
0
Reputation points
commented 2024-02-26T08:26:39.89+00:00
PRADEEPCHEEKATLA-MSFT
76,586
Reputation points Microsoft Employee
1 answer
The User Assigned Identity not appearing when creating an HDI cluster
How can I select my User Assigned Identity when creating an HDI cluster with storage lake gen2 storage? Despite creating the appropriate identity and attaching it to the storage lake with the 'Storage Blob Data Owner' permission, nothing shows up in the…
asked 2024-02-15T18:24:42.0866667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(115.19999999999999, 67%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
Szk231
0
Reputation points
answered 2024-02-16T06:33:13.94+00:00
Izmir Sadigov
0
Reputation points MVP
1 answer
HDInsight Kafka Clusters certificate expire every month or do we have any Maintenace activity's will happen in HDInsight kafka clusters in Azure
We recently created Azure HDInsight kafka clusters. Just want to know did Azure do any maintenace activity's like Certificate renews or any security patching. as we are new to use these services. just want to know about that. Thank you.
asked 2024-01-01T20:34:09.44+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(22.400000000000002, 72%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKS%3C/text%3E%3C/svg%3E)
kamalakuri siva sankara
0
Reputation points
commented 2024-01-29T11:17:24.49+00:00
AnnuKumari-MSFT
30,361
Reputation points Microsoft Employee
1 answer
Unable to register HD Insights resource provider in my Azure Subscription
I'm trying to create a resource group via a terraform script an am unable to. The HD Insight resource provider is stuck at the registering step. So I deleted the subscription. Created a new one and manually tried to register HD Insights and it was still…
asked 2023-10-16T01:00:35.0566667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(172.8, 18%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMS%3C/text%3E%3C/svg%3E)
Medcan Support
5
Reputation points
commented 2023-10-26T09:32:29.4133333+00:00
PRADEEPCHEEKATLA-MSFT
76,586
Reputation points Microsoft Employee
1 answer
Why is the Resource provider Microsoft.HDInsight stuck on Registering?
From the Azure portal I can see the Resource provider Microsoft.HDInsight is stuck on "Registering" status and has been for a couple of hours since I first looked. I've tried to unregister and register again but that just times out. My account…
asked 2023-10-14T10:27:21.81+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(188.79999999999998, 33%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPE%3C/text%3E%3C/svg%3E)
Phil Evans
0
Reputation points
commented 2023-10-19T09:30:49.7533333+00:00
PRADEEPCHEEKATLA-MSFT
76,586
Reputation points Microsoft Employee
4 answers
Registering a HDInsight resource provider never completes
Hi Trying to create an HDInsight cluster but the Microsoft.HDInsight resource provider was not registered. I click on register and left the process overnight. This morning it still showing "Registering". Can anyone help?
asked 2021-06-01T15:03:30.737+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(259.20000000000005, 22%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECR%3C/text%3E%3C/svg%3E)
Carlos Restrepo
1
Reputation point Microsoft Employee
edited the question 2023-10-17T00:08:29.9066667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 9%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKM%3C/text%3E%3C/svg%3E)
KranthiPakala-MSFT
46,422
Reputation points Microsoft Employee
1 answer
How will Azure Application Insight help to optimize my application
Hello ! I would like to know how will enabling Azure application insight help me to analyze the performance issues in my application . WIll enabling this slow down my application / is there anything we need to check before this is enabled. Thanks
asked 2023-10-12T09:36:42.2133333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(86.4, 67%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKB%3C/text%3E%3C/svg%3E)
kavipriya balaji
0
Reputation points
answered 2023-10-12T18:56:46.1966667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(54.400000000000006, 38%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EQ%3C/text%3E%3C/svg%3E)
QuantumCache
20,031
Reputation points
1 answer
How to read and write data to hbase hdinsights with databricks
I'm doing a proof of concept to compare which tool best suits our company, Cosmos DB or A Hbase cluster on HDInsights: I'm trying to read and write data using Databricks on HDinsights HBase. I tried to use the SHC lib but it is only available for version…
asked 2023-09-21T14:38:31.71+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(217.60000000000002, 92%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENI%3C/text%3E%3C/svg%3E)
Nobuyoshi Ishizuka
0
Reputation points
commented 2023-09-25T19:28:50.6966667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(86.4, 36%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBM%3C/text%3E%3C/svg%3E)
BhargavaGunnam-MSFT
25,876
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Azure HD Insights Spot instances.
How to use the Azure HDInsights Spot instances. What is the max limit here?
asked 2023-09-11T17:50:23.2666667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(128, 88%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EH7%3C/text%3E%3C/svg%3E)
HariKrishna-7067
120
Reputation points
accepted 2023-09-12T17:01:28.91+00:00
HariKrishna-7067
120
Reputation points
2 answers
One of the answers was accepted by the question author.
Suggestions for Optimizing Data Processing Speed and Migrating to Azure
Problem Description: We are currently facing a data processing challenge: our data processing workflow runs Java programs on a single machine, and data is stored in a DB server with a total dataset of over a hundred million records. Each run of our data…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(284.8, 87%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
1 answer
Remotly connect to AzureHDInsight 5.0 spark cluster
Hi, I have a local jupyter notebook and Im trying to connect to an AzureHDInsight 5.0 spark cluster. I found tutorials on how to create a jupyter notebook directly inside the Azure Cloud Interface of the cluster, but what if I want to connect remotly? …
asked 2023-08-29T12:42:49.91+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(19.2, 24%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDB%3C/text%3E%3C/svg%3E)
Dario Bertolino
0
Reputation points
commented 2023-09-01T21:25:39.9933333+00:00
KranthiPakala-MSFT
46,422
Reputation points Microsoft Employee
1 answer
HDinsight 4.0 to 5.0
Differences between HDinsight Componant functionality of 4.0 and 5.0 major change , minor change Does the Ambari page affect usability?
asked 2023-08-18T10:20:04.28+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(291.2, 79%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPK%3C/text%3E%3C/svg%3E)
PANUWAT KLAHAN
0
Reputation points
commented 2023-08-30T10:53:36.19+00:00
ShaikMaheer-MSFT
37,896
Reputation points Microsoft Employee
1 answer
How is headnode failure handled in hdinsight ?
Hi, As HDInsight have fixed number of head nodes i.e 2. I'm curious how head node failures are handled in HDinsight ? Does failed headnode gets removed and a new headnode gets added in cluster ? failed headnode remains in cluster ? Thanks, Akshit Mehta
asked 2023-08-11T06:20:42.81+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(105.60000000000001, 21%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3E3%3C/text%3E%3C/svg%3E)
31610895
20
Reputation points
commented 2023-08-29T05:47:47.12+00:00
QuantumCache
20,031
Reputation points
1 answer
One of the answers was accepted by the question author.
Does HDInsight manage zookeeper node ? How are failures handled in zookeeper nodes ?
Hi, First question - As HDInsight have fixed number of zookeeper nodes i.e 3. I'm curious how zookeeper node failures are handled in HDinsight ? Does failed zookeeper node gets removed and a new zookeeper node gets added in cluster ? failed…
asked 2023-08-16T10:08:51.13+00:00
31610895
20
Reputation points
commented 2023-08-22T06:52:45.5433333+00:00
PRADEEPCHEEKATLA-MSFT
76,586
Reputation points Microsoft Employee
2 answers
How to use spot instances with HDInsight
Hi, Is there any way to use Spot VMs in HDInsight ? How to determine which VMs are spot VMs in this list - https://azure.microsoft.com/en-in/pricing/details/virtual-machines/linux/ ? Is there any max limit on number of Spot VMs that can be…
asked 2023-08-16T10:26:35.41+00:00
31610895
20
Reputation points
commented 2023-08-17T06:34:57.5766667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(211.20000000000002, 45%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESZ%3C/text%3E%3C/svg%3E)
shlim@zenithn.com
1
Reputation point
1 answer
How to implement CI/CD on HDInsight ?
Hi, I need recommendations about how to apply CI/CD in HDinsight. I want to create "dev" and "prod" cluster. One Spark and one Hive IQ each. Let's…
asked 2023-07-26T15:51:14.8133333+00:00
Federico Sardo
91
Reputation points
commented 2023-08-09T13:37:17.67+00:00
QuantumCache
20,031
Reputation points