Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,038 questions
0 answers
I/O operations with Azure Databricks REST Jobs API
I have experienced problems with the delivery of arguments via Jobs API. I've outlined the experienced problems in details on Stack Overflow: https://stackoverflow.com/questions/62758094/i-o-operations-with-azure-databricks-rest-jobs-api I would…
asked 2020-07-06T15:38:42.997+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(179.20000000000002, 72%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGM%3C/text%3E%3C/svg%3E)
Galas, Michal
1
Reputation point
commented 2020-07-13T17:59:33.667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 26%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HimanshuSinha-msft
19,386
Reputation points Microsoft Employee
1 answer
Machine Learning Model Deployment
I am new to ML model and am researching using Azure Databricks and MLFlow to train a model. My question is once the model is created, is there a way to host the model that can be downloaded and inferenced remotely ? I am looking for options other than…
asked 2020-07-01T16:26:17.527+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(89.60000000000001, 70%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMS%3C/text%3E%3C/svg%3E)
Mahesh Sivan
1
Reputation point
commented 2020-07-13T12:45:09.187+00:00
romungi-MSFT
43,616
Reputation points Microsoft Employee
1 answer
Azure Web Application with computationally intenstive tasks in Dask and Tensorflow
Hello, I'm developing a data analysis tool for the processing of data from Hydrogen-Deuterium exchange mass spectrometry. We would like to accompany our publication with a deployment of the code on Microsoft Azure so that other researchers can quickly…
asked 2020-06-24T10:22:29.26+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(48, 5%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJS%3C/text%3E%3C/svg%3E)
Jochem Smit
1
Reputation point
answered 2020-07-08T05:16:30.38+00:00
brtrach-MSFT
15,701
Reputation points Microsoft Employee
0 answers
Spark Connector in ADF
Hi, I have created a spark connector to connect to azure data bricks. In copy activity source is spark connector and sink is Azure SQL DB. In spark Connector query, CreatedDate is being converted to String and throwing error where as it is timestamp…
asked 2020-06-19T16:53:27.433+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(64, 74%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EM%3C/text%3E%3C/svg%3E)
Mounica
1
Reputation point
commented 2020-07-07T18:22:07.117+00:00
HimanshuSinha-msft
19,386
Reputation points Microsoft Employee
0 answers
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 7.0 failed 4 times, most recent failure: java.lang.NoClassDefFoundError: Could not initialize class
Hi, I am getting this error despite defining the class. When I execute the notebook first time it works fine but when I execute the same notebook without code change it started throwing this error. As per the error class not defined but trust me class…
asked 2020-06-23T23:20:13.487+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(22.400000000000002, 49%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERK%3C/text%3E%3C/svg%3E)
Rajaniesh Kaushikk
476
Reputation points
commented 2020-07-02T22:39:19.027+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 9%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKM%3C/text%3E%3C/svg%3E)
KranthiPakala-MSFT
46,437
Reputation points Microsoft Employee
0 answers
data bricks scala : data frame column endoing from UTF 8 to windows 1252
HI I am working with data bricks where i have the data in parque and i am generating smaller files out of it , i have a column in this which is string and it has different characters and i have to encode this string value to windows 1252 or windows…
asked 2020-06-22T20:29:26.43+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(150.4, 88%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EM%3C/text%3E%3C/svg%3E)
ManojMathe
1
Reputation point
commented 2020-06-25T22:21:25.563+00:00
HimanshuSinha-msft
19,386
Reputation points Microsoft Employee
1 answer
Third party Python package installed on Databricks cluster gives different results than other Python stacks
We get a Python package developed by a third party. The package implements a standard mathematical model, no machine learning, no randomization. The model turned out to return incorrect results when installed on a Databricks cluster. We tried different…
asked 2020-06-18T07:56:40.197+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(118.4, 82%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHG%3C/text%3E%3C/svg%3E)
Hans Geurtsen
1
Reputation point
commented 2020-06-23T12:14:01.153+00:00
Hans Geurtsen
1
Reputation point
1 answer
Databricks Notebook Activity parameter problem
I feel this is a bug but not sure if it is with ADF or Databricks. I am running a notebook using ADF notebook activity. My notebook has a widget for which I pass the value from ADF. As I need to manually enter the parameter name while configuring…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(25.6, 95%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ET%3C/text%3E%3C/svg%3E)
2 answers
One of the answers was accepted by the question author.
Spark SQL How to get the 5th column from the Spark SQL Query
Hi, I have a headerless file which I am reading in the spark.read to create a data frame now I want to get the value of the 5th column from the file.File is comma seperated. How to achieve it. I know it is possible in the T-SQL but not sure how to…
asked 2020-06-15T13:48:04.437+00:00
Rajaniesh Kaushikk
476
Reputation points
commented 2020-06-16T18:28:18.347+00:00
Rajaniesh Kaushikk
476
Reputation points
2 answers
One of the answers was accepted by the question author.
SparkException: Job aborted due to stage failure: Task not serializable: java.io.NotSerializableException:
Hi, I am running this code but this is throwing this error: SparkException: Job aborted due to stage failure: Task not serializable: java.io.NotSerializableException:
asked 2020-06-14T03:55:51.703+00:00
Rajaniesh Kaushikk
476
Reputation points
accepted 2020-06-16T09:55:30.627+00:00
Rajaniesh Kaushikk
476
Reputation points
0 answers
Azure Databricks - Split column based on special characters in Databricks
I have a column in my csv file that possibly has value in below formats. "Q1_1__Value_-_10_counts" "Value_10_counts" "Q1_1__1__value_yes" This has to be split as below respectively "Value_-_10_counts" …
asked 2020-06-08T10:37:46.403+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(236.8, 80%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJ%3C/text%3E%3C/svg%3E)
Jothi
11
Reputation points
commented 2020-06-15T20:24:28.467+00:00
HimanshuSinha-msft
19,386
Reputation points Microsoft Employee
1 answer
More convenient service to read avro files from Azure Data Lake Gen2
Hi, I have to read lots of avro files created by an Event Hub Capture in a Data Lake Gen2. Data must be filtered, processed and then applied to train a machine learning model. I'm considering Azure Databricks and the Azure Machine Learning service…
asked 2020-06-08T22:37:56.897+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(44.800000000000004, 92%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAC%3C/text%3E%3C/svg%3E)
Ariel Cedola
21
Reputation points
commented 2020-06-15T20:20:36.227+00:00
HimanshuSinha-msft
19,386
Reputation points Microsoft Employee
3 answers
One of the answers was accepted by the question author.
Azure IoT - Query Data from IoT Files
Hello, I am using Azure (Azure Databricks, IoT Hub) to stream unstructured data from IoT devices (i.e. wind turbine), in the form of thousands of files with millions of data captured over a period of 10 years. How do I extract a variety of metadata…
asked 2020-06-15T00:51:10.89+00:00
Sarosh Niazi
21
Reputation points
accepted 2020-06-15T19:37:26.247+00:00
Sarosh Niazi
21
Reputation points
2 answers
File(filePath).exists does not work in Azure databricks
Hi, How to find if file exists in a path in the data lake? Regards Rajaniesh
asked 2020-06-11T20:12:45.693+00:00
Rajaniesh Kaushikk
476
Reputation points
answered 2020-06-14T04:01:39.583+00:00
Rajaniesh Kaushikk
476
Reputation points
2 answers
One of the answers was accepted by the question author.
Accessing dataframe created in Scala from Python command
Is there a way to create a Spark dataframe in Scala command, and then access it in Python, without explicitly writing it to disk and re-reading? In Databricks I can do in Scala dfFoo.createOrReplaceTempView("temp_df_foo") and it then in…
1 answer
One of the answers was accepted by the question author.
Standard Configuration Conponents of the Azure Datacricks
Hello, Could you please tell me standard configuration components of the Azure Databricks. What are the Azure components (storage?) required for the configuration of the Azure Databricks? Thank you. Sincerely, Kenjiro Majima
asked 2020-05-29T04:20:47.017+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(102.4, 60%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKM%3C/text%3E%3C/svg%3E)
Kenjiro Majima
21
Reputation points
commented 2020-06-03T05:07:39.423+00:00
Kenjiro Majima
21
Reputation points
1 answer
One of the answers was accepted by the question author.
How to integrate/add more metrics & info into Ganglia UI in Databricks Jobs
As per https://learn.microsoft.com/en-us/azure/databricks/clusters/clusters-manage#monitor-performance, Ganglia metrics Collection Period Snapshot modifications can be done using init scripts. Could you please help with pointers to modify by default…
asked 2020-05-08T08:37:57.927+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(128, 57.99999999999999%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERH%3C/text%3E%3C/svg%3E)
Ramya Harinarthini_MSFT
5,311
Reputation points Microsoft Employee
accepted 2020-05-08T09:13:16.457+00:00
Ramya Harinarthini_MSFT
5,311
Reputation points Microsoft Employee
2 answers
One of the answers was accepted by the question author.
Azure databricks is not available in free trial subscription
If i have understood it right, Azure databricks is not available on free tier account. I currently have a free tier, 12 month subscription. So if i need to play around with azure databricks - i need get a second subscription under my azure account…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(278.4, 35%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)