212 questions with Azure HDInsight tags

Sort by: Updated
1 answer

Difference between synapse and hdinsight

What are major difference between synapse and HD insight and under what circumstances one should use synapse and vice versa

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,988 questions
Azure HDInsight
Azure HDInsight
An Azure managed cluster service for open-source analytics.
212 questions
asked 2021-10-12T12:43:25.457+00:00
Shivam Ramola 71 Reputation points
commented 2024-11-06T12:09:12.1433333+00:00
Michael Schmidt 80 Reputation points
1 answer

Spark HDInsight cluster deployment fails with an internal server error

I've tried deploying a 5.1.3 HDInsight cluster twice, and both times it fails with an internal server error. JSON below. I chose all the defaults except for using an existing storage account and virtual network. Any suggestions appreciated. { …

Azure HDInsight
Azure HDInsight
An Azure managed cluster service for open-source analytics.
212 questions
asked 2024-10-27T20:54:05.21+00:00
Michael P Spertus 0 Reputation points
commented 2024-10-30T13:16:19.84+00:00
Smaran Thoomu 16,730 Reputation points Microsoft Vendor
0 answers

Azure HDInsight: Managing Unhealthy Nodes and Sessions

Hi Team, We are currently facing an issue with Unhealthy Nodes on Azure HDInsight. We have one centralized HDInsight cluster shared between Data Teams, where multiple Spark and Hive jobs are being submitted continuously. However, due to either overload…

Azure HDInsight
Azure HDInsight
An Azure managed cluster service for open-source analytics.
212 questions
asked 2024-10-21T10:38:47.1333333+00:00
Mahesh Sanga 0 Reputation points Microsoft Employee
commented 2024-10-24T19:54:07.18+00:00
Chandra Boorla 2,605 Reputation points Microsoft Vendor
1 answer

spark notebooks with terraform

i want to create notebooks using automation for automation i am using terraform, can we invoke spark notebooks in terraform

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,988 questions
Azure HDInsight
Azure HDInsight
An Azure managed cluster service for open-source analytics.
212 questions
asked 2024-08-28T17:00:07.9466667+00:00
Gnana Sai krishna 80 Reputation points
commented 2024-09-02T08:59:22.7133333+00:00
PRADEEPCHEEKATLA-MSFT 90,221 Reputation points Microsoft Employee
1 answer

Stucking in Creating HDinsight Clusters

Hi, My regional quota for vCPUs is 20, but I'm still unable to create the HDInsight clusters. Can anyone help me?

Azure HDInsight
Azure HDInsight
An Azure managed cluster service for open-source analytics.
212 questions
asked 2024-08-14T20:09:43.2133333+00:00
Ziwei Zeng 10 Reputation points
answered 2024-08-16T11:40:40.8966667+00:00
phemanth 11,200 Reputation points Microsoft Vendor
1 answer

Unable to run Spark activity through data factory pipeline

After configuring the network and ensuring that the test connection between data factory and the respective spark cluster is successful, I am unable to submit nor view any spark jobs that are submitted through the data pipeline using the spark activity.…

Azure HDInsight
Azure HDInsight
An Azure managed cluster service for open-source analytics.
212 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,826 questions
asked 2024-07-31T10:23:06.4366667+00:00
Aleksandr Maxermis 0 Reputation points
edited a comment 2024-08-06T15:14:11.93+00:00
Smaran Thoomu 16,730 Reputation points Microsoft Vendor
0 answers

Kafka REST proxy endpoint is corrupting and wrongly parsing data

Hello Team, I recently activated REST proxy endpoint for HdInsight kafka. The endpoint is used to produce data to a topic, and upon consuming from the topic I observed that the data is wrongly encoded and parsed. If I try to produce data from the client…

Azure HDInsight
Azure HDInsight
An Azure managed cluster service for open-source analytics.
212 questions
asked 2024-07-11T20:33:51.8733333+00:00
AwesomePaul 20 Reputation points
commented 2024-07-22T05:02:13.68+00:00
PRADEEPCHEEKATLA-MSFT 90,221 Reputation points Microsoft Employee
1 answer

How can we integrate hadoop logs in insights

Hi , Need to integrage spark application logs in azure insights or api insgihts. And also need to build monitoring dashboard for azure spark hid ingishts Any inputs please

Azure HDInsight
Azure HDInsight
An Azure managed cluster service for open-source analytics.
212 questions
asked 2024-07-17T11:11:18.2966667+00:00
Panneeru, Vikram 0 Reputation points
answered 2024-07-17T17:34:56.18+00:00
Bhargava-MSFT 31,116 Reputation points Microsoft Employee
1 answer One of the answers was accepted by the question author.

Unable to use HdinsightKafka REST proxy API after authenticating with access token.

Hello Team, I'm having issues connecting to HdInsight kafka using REST proxy endpoint. I followed the steps here (https://learn.microsoft.com/en-us/azure/hdinsight/kafka/rest-proxy), and was able to get token and authenticate the endpoint. However,…

Azure HDInsight
Azure HDInsight
An Azure managed cluster service for open-source analytics.
212 questions
asked 2024-07-10T12:25:43.3633333+00:00
AwesomePaul 20 Reputation points
commented 2024-07-12T08:35:26.7166667+00:00
AwesomePaul 20 Reputation points
1 answer

Kafka REST Proxy Authentication

We would like to develop a HDInsight Kafka cluster to share real time data with a subcontractor. The REST proxy documentation indicates that "Kafka clients that need access to the REST proxy should be registered to a group by the group owner."…

Azure HDInsight
Azure HDInsight
An Azure managed cluster service for open-source analytics.
212 questions
asked 2022-02-18T18:30:46.19+00:00
Mike McNulty 1 Reputation point
commented 2024-07-10T10:15:39.4366667+00:00
AwesomePaul 20 Reputation points
1 answer

Unclear what generates costs

After requesting a quota increase I managed to create a HDInsight cluster. The support told me that services generate costs only when they are in use. According to this page https://learn.microsoft.com/en-us/azure/hdinsight/hdinsight-delete-cluster I…

Azure HDInsight
Azure HDInsight
An Azure managed cluster service for open-source analytics.
212 questions
asked 2024-07-05T14:13:40.68+00:00
blubb-5305 20 Reputation points
commented 2024-07-08T22:11:19.6433333+00:00
Bhargava-MSFT 31,116 Reputation points Microsoft Employee
1 answer One of the answers was accepted by the question author.

HDInsight Version 3.6 is Deprecated,User SubscriptionId 'XXX' does not have cores left to create resource 'hdi-kafka'. Required: 38, Available: 0.

I am going through this tutorial and getting this error when trying to create a HDInsight cluster: HDInsight Version 3.6 is Deprecated,User SubscriptionId 'XXX' does not have cores left to create resource 'hdi-kafka'. Required: 38, Available: 0. This…

Azure HDInsight
Azure HDInsight
An Azure managed cluster service for open-source analytics.
212 questions
asked 2024-07-02T09:00:09.9866667+00:00
blubb-5305 20 Reputation points
accepted 2024-07-05T13:38:15.2733333+00:00
blubb-5305 20 Reputation points
4 answers

Unable to create HDInsight cluster using Microsoft tutorial

I am a student trying to learn how to setup HDInsight Hadoop cluster. I have signed up and received the free $200 credit. I am following the steps laid out in Microsoft's tutorial here:…

Azure HDInsight
Azure HDInsight
An Azure managed cluster service for open-source analytics.
212 questions
asked 2020-12-21T18:46:02.527+00:00
Justin Birchard 1 Reputation point
edited a comment 2024-07-01T10:50:46.2933333+00:00
blubb-5305 20 Reputation points
1 answer

How to restart the Kafka services on HDInsight Kafka Cluster

Im trying to restart the kafka services which were configured on HDInsight kafka cluster as below. But they are getting timeout. Please advise how to proceed. n0-kafkah:/usr/hdp/4.1.18.1/kafka/bin$ sudo ./kafka-server-start.sh…

Azure HDInsight
Azure HDInsight
An Azure managed cluster service for open-source analytics.
212 questions
asked 2024-06-18T09:55:39.4166667+00:00
Kommana, Swathi 0 Reputation points
edited an answer 2024-06-25T08:23:43.53+00:00
PRADEEPCHEEKATLA-MSFT 90,221 Reputation points Microsoft Employee
1 answer

Active worker nodes usage in HDinsight cluster for the period of last one year

Hi Team, Can someone help us to get the Active worker nodes usage in HDinsight cluster for the period of last one year. Thanks & Regards, Udaya

Azure AI Metrics Advisor
Azure AI Metrics Advisor
An Azure artificial intelligence analytics service that proactively monitors metrics and diagnoses issues.
83 questions
Azure HDInsight
Azure HDInsight
An Azure managed cluster service for open-source analytics.
212 questions
asked 2022-01-24T12:50:17.057+00:00
UDAYA SRINIVASARAO KOTHAMASU 1 Reputation point
commented 2024-06-20T13:36:43.6766667+00:00
golla mamatha 0 Reputation points
1 answer One of the answers was accepted by the question author.

Why does HDInsight quota show 0 available cores?

When I view my quotas, I am not using any vCores, and it says I have 36 cores available. But when I try to set up a HDInsight Spark Cluster in the West Europe region it says I have 0 available. I have a pay-as-you basic subscription.

Azure HDInsight
Azure HDInsight
An Azure managed cluster service for open-source analytics.
212 questions
asked 2024-06-12T16:52:39.5466667+00:00
NLE87 20 Reputation points
accepted 2024-06-13T11:56:19.3933333+00:00
NLE87 20 Reputation points
1 answer

How to implement CI/CD on HDInsight ?

Hi, I need recommendations about how to apply CI/CD in HDinsight. I want to create "dev" and "prod" cluster. One Spark and one Hive IQ each. Let's…

Azure HDInsight
Azure HDInsight
An Azure managed cluster service for open-source analytics.
212 questions
asked 2023-07-26T15:51:14.8133333+00:00
Federico Sardo 91 Reputation points
commented 2024-06-06T17:01:15.9966667+00:00
Federico Sardo 91 Reputation points
1 answer One of the answers was accepted by the question author.

Find activity logs sent to a log analytics workspace via a diagnostic setting

I am trying to export the activity logs of an HDInsight cluster, to a log anaytics workspace via a diagnostic setting. But I am unable to find the exported logs. I tried using the following query: search * | where TimeGenerated == "z" How…

Azure Monitor
Azure Monitor
An Azure service that is used to collect, analyze, and act on telemetry data from Azure and on-premises environments.
3,315 questions
Azure HDInsight
Azure HDInsight
An Azure managed cluster service for open-source analytics.
212 questions
asked 2024-06-01T04:19:20.5866667+00:00
Khushika Shringi 20 Reputation points Microsoft Intern
edited an answer 2024-06-03T13:48:06.0566667+00:00
Monalla-MSFT 13,031 Reputation points
1 answer

Register button is disabled for Azure HDInsight

Hello, I hope this message finds you well. We are having an issue with the registration button for the HDInsight resource provider in the Azure portal. When trying to register from the account of the user with the global adminitiration role, I noticed…

Azure HDInsight
Azure HDInsight
An Azure managed cluster service for open-source analytics.
212 questions
asked 2024-04-26T07:52:04.8166667+00:00
eyma-6141 0 Reputation points
commented 2024-05-02T03:38:04.0933333+00:00
PRADEEPCHEEKATLA-MSFT 90,221 Reputation points Microsoft Employee
0 answers

Zero cores available to create Hdinsight cluster

I am using West Central Germany time zone to create a very small hd-insight Hadoop cluster but there are zero cores available. I have open a ticket with the tech support but still no quota allocated. I really need this to be done. Please help

Azure HDInsight
Azure HDInsight
An Azure managed cluster service for open-source analytics.
212 questions
asked 2024-04-23T09:20:00.4966667+00:00
arbab nasir 0 Reputation points
commented 2024-04-25T05:37:57.53+00:00
PRADEEPCHEEKATLA-MSFT 90,221 Reputation points Microsoft Employee