Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,070 questions
2 answers
One of the answers was accepted by the question author.
Spark SQL How to get the 5th column from the Spark SQL Query
Hi, I have a headerless file which I am reading in the spark.read to create a data frame now I want to get the value of the 5th column from the file.File is comma seperated. How to achieve it. I know it is possible in the T-SQL but not sure how to…
asked 2020-06-15T13:48:04.437+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(22.400000000000002, 49%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERK%3C/text%3E%3C/svg%3E)
Rajaniesh Kaushikk
476
Reputation points
commented 2020-06-16T18:28:18.347+00:00
Rajaniesh Kaushikk
476
Reputation points
2 answers
One of the answers was accepted by the question author.
SparkException: Job aborted due to stage failure: Task not serializable: java.io.NotSerializableException:
Hi, I am running this code but this is throwing this error: SparkException: Job aborted due to stage failure: Task not serializable: java.io.NotSerializableException:
asked 2020-06-14T03:55:51.703+00:00
Rajaniesh Kaushikk
476
Reputation points
accepted 2020-06-16T09:55:30.627+00:00
Rajaniesh Kaushikk
476
Reputation points
0 answers
Azure Databricks - Split column based on special characters in Databricks
I have a column in my csv file that possibly has value in below formats. "Q1_1__Value_-_10_counts" "Value_10_counts" "Q1_1__1__value_yes" This has to be split as below respectively "Value_-_10_counts" …
asked 2020-06-08T10:37:46.403+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(236.8, 80%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJ%3C/text%3E%3C/svg%3E)
Jothi
11
Reputation points
commented 2020-06-15T20:24:28.467+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 26%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HimanshuSinha-msft
19,381
Reputation points Microsoft Employee
1 answer
More convenient service to read avro files from Azure Data Lake Gen2
Hi, I have to read lots of avro files created by an Event Hub Capture in a Data Lake Gen2. Data must be filtered, processed and then applied to train a machine learning model. I'm considering Azure Databricks and the Azure Machine Learning service…
asked 2020-06-08T22:37:56.897+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(44.800000000000004, 92%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAC%3C/text%3E%3C/svg%3E)
Ariel Cedola
21
Reputation points
commented 2020-06-15T20:20:36.227+00:00
HimanshuSinha-msft
19,381
Reputation points Microsoft Employee
3 answers
One of the answers was accepted by the question author.
Azure IoT - Query Data from IoT Files
Hello, I am using Azure (Azure Databricks, IoT Hub) to stream unstructured data from IoT devices (i.e. wind turbine), in the form of thousands of files with millions of data captured over a period of 10 years. How do I extract a variety of metadata…
asked 2020-06-15T00:51:10.89+00:00
Sarosh Niazi
21
Reputation points
accepted 2020-06-15T19:37:26.247+00:00
Sarosh Niazi
21
Reputation points
2 answers
File(filePath).exists does not work in Azure databricks
Hi, How to find if file exists in a path in the data lake? Regards Rajaniesh
asked 2020-06-11T20:12:45.693+00:00
Rajaniesh Kaushikk
476
Reputation points
answered 2020-06-14T04:01:39.583+00:00
Rajaniesh Kaushikk
476
Reputation points
2 answers
One of the answers was accepted by the question author.
Accessing dataframe created in Scala from Python command
Is there a way to create a Spark dataframe in Scala command, and then access it in Python, without explicitly writing it to disk and re-reading? In Databricks I can do in Scala dfFoo.createOrReplaceTempView("temp_df_foo") and it then in…
1 answer
One of the answers was accepted by the question author.
Standard Configuration Conponents of the Azure Datacricks
Hello, Could you please tell me standard configuration components of the Azure Databricks. What are the Azure components (storage?) required for the configuration of the Azure Databricks? Thank you. Sincerely, Kenjiro Majima
asked 2020-05-29T04:20:47.017+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(102.4, 60%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKM%3C/text%3E%3C/svg%3E)
Kenjiro Majima
21
Reputation points
commented 2020-06-03T05:07:39.423+00:00
Kenjiro Majima
21
Reputation points
1 answer
One of the answers was accepted by the question author.
How to integrate/add more metrics & info into Ganglia UI in Databricks Jobs
As per https://learn.microsoft.com/en-us/azure/databricks/clusters/clusters-manage#monitor-performance, Ganglia metrics Collection Period Snapshot modifications can be done using init scripts. Could you please help with pointers to modify by default…
asked 2020-05-08T08:37:57.927+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(128, 57.99999999999999%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERH%3C/text%3E%3C/svg%3E)
Ramya Harinarthini_MSFT
5,331
Reputation points Microsoft Employee
accepted 2020-05-08T09:13:16.457+00:00
Ramya Harinarthini_MSFT
5,331
Reputation points Microsoft Employee
2 answers
One of the answers was accepted by the question author.
Azure databricks is not available in free trial subscription
If i have understood it right, Azure databricks is not available on free tier account. I currently have a free tier, 12 month subscription. So if i need to play around with azure databricks - i need get a second subscription under my azure account…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(278.4, 35%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)