Azure HDInsight
An Azure managed cluster service for open-source analytics.
199 questions
2 answers
Unable to Create HDInsight(spark) cluster
I am trying to create a HDInsight (spark) cluster, and getting the error message below. Please advise. I have tried all different region and different node size, and I am getting the same error message. You have reached your subscription's cores…
asked 2021-06-16T15:29:28.78+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 72%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMG%3C/text%3E%3C/svg%3E)
Megha Gupta
1
Reputation point
commented 2021-06-21T12:39:06.613+00:00
PRADEEPCHEEKATLA-MSFT
77,751
Reputation points Microsoft Employee
1 answer
Azure HDInsight Subscription Registration
I am currently having trouble w/ registering Microsoft.HDInsight on my subscription. Any recommendations? Do I need to create another subscription? Thanks.
asked 2021-06-05T20:51:09.637+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(147.20000000000002, 44%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJR%3C/text%3E%3C/svg%3E)
James Robert Hizon
26
Reputation points
commented 2021-06-10T03:41:54.077+00:00
PRADEEPCHEEKATLA-MSFT
77,751
Reputation points Microsoft Employee
1 answer
Azure Subscription HDInsight Access
I had to create a new account so I could try to use HDInsight (I actually did not use any credits from current account). I currently am am trying to set up HDInsight cluster but my azure subscription says the following: The subscription is not registered…
asked 2021-06-05T17:10:15.483+00:00
James Robert Hizon
26
Reputation points
commented 2021-06-10T03:41:35.45+00:00
PRADEEPCHEEKATLA-MSFT
77,751
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
HDInsight Cluster Deployment Failure
Hi. I currently am experiencing issues using Azure HDInsight cluster for a Mini-Project through Springboard. I am curious as to why I am receiving the following error after deploying my cluster: …
asked 2021-06-01T22:40:06.483+00:00
James Robert Hizon
26
Reputation points
commented 2021-06-07T02:04:38.09+00:00
PRADEEPCHEEKATLA-MSFT
77,751
Reputation points Microsoft Employee
1 answer
Automatically delete HDInsight cluster after period of inactivity
Hi, I have created an Azure template for creating HDInsight cluster. I would like to be able to set an option to delete the cluster after a period of inactivity. Ideally, I could set up this parameter during the creation and after (let's say) an…
asked 2021-05-31T08:51:12.91+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(60.8, 28.000000000000004%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
suba1987
1
Reputation point
commented 2021-06-04T09:50:29.327+00:00
PRADEEPCHEEKATLA-MSFT
77,751
Reputation points Microsoft Employee
1 answer
HDInsight is spanned over availability zoned in region?
Hi I wonder how HDInsight fault tolerance is working. There are availability zones in each region, so I believe HDInsight's head nodes and worker nodes are spanned over these availability zones, but it's true? No matter how hard the failover works,…
asked 2021-05-20T04:07:14.94+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(214.4, 73%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EY%3C/text%3E%3C/svg%3E)
yu
21
Reputation points
commented 2021-05-24T10:29:25.767+00:00
PRADEEPCHEEKATLA-MSFT
77,751
Reputation points Microsoft Employee
1 answer
When HDInsight worker node is stopped, the node will be automatically rebooted?
Hello As I described in this title, when HDInsight worker node is stopped for some reason(not for Azure infrastructure issue, but for the issue inside worker nodes stems from such as CPU or memory issues), the worker node will be automatically…
asked 2021-05-20T03:50:55.463+00:00
yu
21
Reputation points
commented 2021-05-24T10:29:03.743+00:00
PRADEEPCHEEKATLA-MSFT
77,751
Reputation points Microsoft Employee
1 answer
HDInsight cluster(Hadoop) creation problem
Hi, I need to use HDInsight sets(hadoop) for my homework for a short time. I called all the sources, but I couldn't upload them. I made a zone change for the solution did not work, I wrote 1 in the node field, but the problem was not solved, I think…
asked 2021-05-16T00:06:12.65+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(0, 40%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EEP%3C/text%3E%3C/svg%3E)
Eyüp
1
Reputation point
commented 2021-05-24T03:59:06.243+00:00
PRADEEPCHEEKATLA-MSFT
77,751
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Cannot find and add the application 'Starburst Presto' to HDInsight, but how to install it?
Hi I'm new to Azure and trying to create HDInsight cluster. I would like to use presto on HDInsight. When creating a Hadoop cluster, HDInsight v4.0, and I tried to add the application Starburst Presto to the cluster on Configuration + pricing tab, …
asked 2021-05-15T21:46:09.827+00:00
yu
21
Reputation points
commented 2021-05-20T05:38:03.743+00:00
PRADEEPCHEEKATLA-MSFT
77,751
Reputation points Microsoft Employee
0 answers
Not able to limit the size of STDERR and STDOUT logs for Azure HDInsight deployed Spark Application!!!
I have tried editing the log4j properties file in https://CLUSTERNAME.azurehdinsight.net/#/main/services/SPARK2/configs, as mentioned in the following link: https://learn.microsoft.com/en-us/azure/hdinsight/spark/troubleshoot-debug-wasb I have done…
asked 2021-04-20T09:16:53.993+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(217.60000000000002, 59%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGA%3C/text%3E%3C/svg%3E)
Goel, Akanksha
66
Reputation points
commented 2021-05-04T04:24:45.77+00:00
PRADEEPCHEEKATLA-MSFT
77,751
Reputation points Microsoft Employee
1 answer
what is the Livy versions of all spark versions in HDInsight
Hi，I want to know the Livy version of different spark version in HDInsight. all the spark version is the pic below.
asked 2021-04-28T02:07:49.59+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(0, 98%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ELZ%3C/text%3E%3C/svg%3E)
Liupeng Zhuang
1
Reputation point
commented 2021-04-30T10:16:15.92+00:00
PRADEEPCHEEKATLA-MSFT
77,751
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Spark metrics on HDInsight
Spark has a configurable metrics system based on the Dropwizard Metrics Library. This allows users to report Spark metrics to a variety of sinks including HTTP, JMX, and CSV files. I know that Spark on HDInsight can be connected to a Log Analytics…
asked 2021-04-14T07:59:52.747+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(64, 94%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESS%3C/text%3E%3C/svg%3E)
Sebastian Schmidt
21
Reputation points
accepted 2021-04-20T13:30:29.18+00:00
Sebastian Schmidt
21
Reputation points
1 answer
Having issue installing mysql-connector-python package to Jupiter notebook on PySpark3 using Script Action-script failing
Here is my Script Action commands: !/usr/bin/env bash sudo /usr/bin/anaconda/bin/conda create --prefix /usr/bin/anaconda/envs/bgdatamysql python=3.5 anaconda --yes sudo /usr/bin/anaconda/bin/conda install -c anaconda mysql-connector-python -n…
asked 2021-03-08T16:47:56.203+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(240, 82%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAO%3C/text%3E%3C/svg%3E)
Alphonse Okossi
141
Reputation points
commented 2021-04-12T11:57:16.007+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(86.4, 61%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERR%3C/text%3E%3C/svg%3E)
Raghu Ram Mahankali (Quadrant Resource)
1
Reputation point
3 answers
One of the answers was accepted by the question author.
Data Type and Format In External table PolyBase
Hello there, I am trying to query Azure Storage Account CDM csv file from SQL server on-prem. I created external table with all the data type as varchar. However, I need to display date time as datetime and need to remove double quote from string…
asked 2021-04-07T16:49:09.937+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(227.2, 5%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPA%3C/text%3E%3C/svg%3E)
Palash Aich
21
Reputation points
accepted 2021-04-12T06:53:02.917+00:00
Palash Aich
21
Reputation points
0 answers
Outbound ports to HDInsight management
Hello, What are the required ports for outbound connection of HDInsight to management IPs and any other needed Azure services when configured with a resource provider connection set to ‘Outbound’? …
asked 2021-03-23T14:03:02.817+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(246.4, 53%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAB%3C/text%3E%3C/svg%3E)
Alex Boata
21
Reputation points
commented 2021-04-09T09:24:47.527+00:00
PRADEEPCHEEKATLA-MSFT
77,751
Reputation points Microsoft Employee
1 answer
Collision on non-unique "headnodehost" hostname across HDInsight clusters
The context: Let's suppose I have multiple HdInsight4.0 clusters. Also suppose that I would like to access the Hadoop services eg jobhistory server running inside these clusters. Let's suppose I get the corresponding jobhistory address from each…
asked 2021-03-03T10:10:10.343+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(128, 8%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
Anonymous
answered 2021-03-29T22:13:24.28+00:00
Anonymous
1 answer
One of the answers was accepted by the question author.
Azure HDInsight HBase - Create non-admin SSH user
I would like to create a standard user account on all HDInsight nodes that is a non-admin account. This will be used to login to the nodes and run some basic commands. There is an admin account created by default on all nodes, but is there a way for…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(64, 61%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EN%3C/text%3E%3C/svg%3E)
1 answer
HWC and Hive (HDinsight): reserved keyword as a column name
with an attempt to save the dataframe for a table which has a column name 'timestamp' and SaveMode.Overwrite, the following exception occurs: org.apache.hadoop.hive.ql.parse.ParseException:line 1:47 cannot recognize input near 'timestamp' 'timestamp'…
asked 2021-03-15T15:30:26.08+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(230.39999999999998, 4%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECV%3C/text%3E%3C/svg%3E)
Core Velocity
1
Reputation point
commented 2021-03-23T04:21:29.887+00:00
PRADEEPCHEEKATLA-MSFT
77,751
Reputation points Microsoft Employee
1 answer
Setting StatusFolder for HDInsight Spark job triggered through Data Factory
Currently our Spark job runs result in a number of folders with random guid names being created in the root directory of the container we use as our HDInsight cluster storage. This seems to be the folder in the context of which the job runs, it has a…
.NET
.NET
Microsoft Technologies based on the .NET software framework.
3,395 questions
Azure HDInsight
Azure HDInsight
An Azure managed cluster service for open-source analytics.
199 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,601 questions
asked 2021-03-11T21:33:49.843+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(179.20000000000002, 54%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESC%3C/text%3E%3C/svg%3E)
Shweta Chandramouli
1
Reputation point
commented 2021-03-23T04:20:01.213+00:00
PRADEEPCHEEKATLA-MSFT
77,751
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Outbound proxy to HDInsight management
Hello, How can we route traffic to HDInsight management through a proxy to avoid opening NSG outbound and also to not use an Azure firewall? https://learn.microsoft.com/en-us/azure/hdinsight/hdinsight-restrict-outbound-traffic Thank you, …
asked 2021-03-18T12:00:04.383+00:00
Alex Boata
21
Reputation points
commented 2021-03-22T08:59:10.227+00:00
Alex Boata
21
Reputation points