Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,070 questions
1 answer
One of the answers was accepted by the question author.
Installing gsl on a Databricks cluster
Hello, I am trying to install the GNU Scientific Library on a Databricks high-concurrency cluster. When I run the following shell script: %sh sudo apt-get install libgsl-dev The script keeps running forever, even though the file is about 8Mb…
asked 2020-07-28T06:35:46.18+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(163.2, 10%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAB%3C/text%3E%3C/svg%3E)
Arko Bose
21
Reputation points
commented 2020-07-29T12:41:07.213+00:00
PRADEEPCHEEKATLA-MSFT
85,346
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Can you help me with listing all the files under mounted blob store?
I have mounted the blob store but I am getting following erorr. Please find attached screenshot.
asked 2020-07-23T09:42:45.263+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(217.60000000000002, 59%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGA%3C/text%3E%3C/svg%3E)
Goel, Akanksha
66
Reputation points
commented 2020-07-29T09:36:42.967+00:00
Goel, Akanksha
66
Reputation points
0 answers
DDL opertaions against Azure Cosmos DB Cassandra API from Spark
Hi, I'm having very strange behavior on DDL operations against Cosmos DB w/ Cassandra API from Spark running in Databricks. When creating a keyspace and table as the following, it does execute the create statements with no errors, BUT it really…
asked 2020-07-07T22:29:17.973+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(150.4, 87%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESH%3C/text%3E%3C/svg%3E)
Sungho Hong
1
Reputation point
commented 2020-07-28T05:37:14.93+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(297.6, 97%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAS%3C/text%3E%3C/svg%3E)
Anurag Sharma
17,586
Reputation points
0 answers
ADF Pipeline error
I have had a pipeline running for the past month on a daily basis. Today the pipeline failed saying that the notebook does not exist. I am able to access the said notebook and it is in the proper location that the Pipeline has defined it and I can…
asked 2020-07-20T22:28:50.037+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(147.20000000000002, 30%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EB%3C/text%3E%3C/svg%3E)
Brittany
1
Reputation point
commented 2020-07-27T08:00:11.84+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(108.80000000000001, 9%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HarithaMaddi-MSFT
10,136
Reputation points
1 answer
How do we add user to synapse analytics workspace ?
Hi, How do i add AAD user to a specific Synapse Analytics Workspace. At the moment it is showing only SQL Pool Admin, Spark Admin and Workspace Admin. I want to just add end user to a workspace so that they can work with it. In Azure Databricks we…
asked 2020-07-13T14:14:27.267+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(275.2, 79%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAD%3C/text%3E%3C/svg%3E)
Anbu.Dhanushkodi
1
Reputation point
commented 2020-07-23T07:49:05.83+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(249.60000000000002, 43%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ER%3C/text%3E%3C/svg%3E)
Rishabh
11
Reputation points
1 answer
One of the answers was accepted by the question author.
Not able to mount Blob store on DBFS
I am not able to mount Blob container on the DBFS path. Please can you help me with this issue.PFA screenshot!
asked 2020-07-22T14:17:59.35+00:00
Goel, Akanksha
66
Reputation points
commented 2020-07-23T06:44:53.683+00:00
Goel, Akanksha
66
Reputation points
1 answer
How to access DBFS datta using REST API
Hi, I want to access DBFS data from outside the databricks using API with the help of python code. How can I do that
asked 2020-07-02T02:27:32.813+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(54.400000000000006, 61%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVS%3C/text%3E%3C/svg%3E)
Vineet Sajwan
1
Reputation point
commented 2020-07-22T10:21:29.397+00:00
PRADEEPCHEEKATLA-MSFT
85,346
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
$log container Name not in ARM template of azure Storage but it is deplyo on Target
HI All, i generate a ARM template for azure storage account- in JSON i didn't find any $log container, but after deployment i found a $log container. I Try to delete this container $log but not able to delete. message- i don't have permission. i…
asked 2020-07-03T09:52:24.953+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(262.40000000000003, 20%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMV%3C/text%3E%3C/svg%3E)
manish verma
441
Reputation points
accepted 2020-07-20T14:34:33.553+00:00
manish verma
441
Reputation points
1 answer
Spark jobs not running in a notebook
I am currently running the "1.Reading Data - CSV" notebook from the "Read and write data in Azure Databricks" module on Microsoft learn. When I tried to run the cell "# A reference to our tab-separated-file", the Spark jobs…
asked 2020-06-16T22:50:34.343+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(25.6, 70%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESY%3C/text%3E%3C/svg%3E)
Sasha Yang
1
Reputation point
commented 2020-07-20T11:30:56.053+00:00
PRADEEPCHEEKATLA-MSFT
85,346
Reputation points Microsoft Employee
2 answers
Upsert data in to SQL from delta table
Hello Team, we have scenario where we have to get the data from lake , process it and then store in SQL database . This is what we are doing Read the entity from Lake Store that in delta table _staging Do merge between delta table and…
asked 2020-06-26T17:49:42.503+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(3.2, 70%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ER%3C/text%3E%3C/svg%3E)
Rocky420
21
Reputation points
commented 2020-07-17T07:57:35.09+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(185.6, 76%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECM%3C/text%3E%3C/svg%3E)
ChiragMishra-MSFT
956
Reputation points
2 answers
Issue in accessing delta table in datalake gen2 storage account with databricks cluster (latest stable version)
Recently, i am encountering an issue in the databricks cluster where it could not accessing the delta table (unmanaged delta table) which parquet files are stored in the azure datalake gen2 storage account. The issue is it could not read/update from the…
asked 2020-06-30T00:14:53.667+00:00
Keat Ooi
31
Reputation points
commented 2020-07-14T23:57:02.437+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 9%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKM%3C/text%3E%3C/svg%3E)
KranthiPakala-MSFT
46,442
Reputation points Microsoft Employee
1 answer
Install third party libraries in Azure Databricks
Hello, I am trying to install a library "pythonnet" in Azure Databricks. I tried installing it through PyPI, through Python Wheel option and also the JAR option. None of these works for me. I need to connect databricks notebook to Azure…
asked 2020-06-25T19:30:03.83+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 43%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESV%3C/text%3E%3C/svg%3E)
Shivani Vyas
21
Reputation points
commented 2020-07-13T21:22:53.083+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(172.8, 70%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
MartinJaffer-MSFT
26,061
Reputation points
0 answers
I/O operations with Azure Databricks REST Jobs API
I have experienced problems with the delivery of arguments via Jobs API. I've outlined the experienced problems in details on Stack Overflow: https://stackoverflow.com/questions/62758094/i-o-operations-with-azure-databricks-rest-jobs-api I would…
asked 2020-07-06T15:38:42.997+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(179.20000000000002, 72%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGM%3C/text%3E%3C/svg%3E)
Galas, Michal
1
Reputation point
commented 2020-07-13T17:59:33.667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 26%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HimanshuSinha-msft
19,381
Reputation points Microsoft Employee
1 answer
Machine Learning Model Deployment
I am new to ML model and am researching using Azure Databricks and MLFlow to train a model. My question is once the model is created, is there a way to host the model that can be downloaded and inferenced remotely ? I am looking for options other than…
asked 2020-07-01T16:26:17.527+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(89.60000000000001, 70%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMS%3C/text%3E%3C/svg%3E)
Mahesh Sivan
1
Reputation point
commented 2020-07-13T12:45:09.187+00:00
romungi-MSFT
43,686
Reputation points Microsoft Employee
1 answer
Azure Web Application with computationally intenstive tasks in Dask and Tensorflow
Hello, I'm developing a data analysis tool for the processing of data from Hydrogen-Deuterium exchange mass spectrometry. We would like to accompany our publication with a deployment of the code on Microsoft Azure so that other researchers can quickly…
asked 2020-06-24T10:22:29.26+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(48, 5%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJS%3C/text%3E%3C/svg%3E)
Jochem Smit
1
Reputation point
answered 2020-07-08T05:16:30.38+00:00
brtrach-MSFT
15,791
Reputation points Microsoft Employee
0 answers
Spark Connector in ADF
Hi, I have created a spark connector to connect to azure data bricks. In copy activity source is spark connector and sink is Azure SQL DB. In spark Connector query, CreatedDate is being converted to String and throwing error where as it is timestamp…
asked 2020-06-19T16:53:27.433+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(64, 74%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EM%3C/text%3E%3C/svg%3E)
Mounica
1
Reputation point
commented 2020-07-07T18:22:07.117+00:00
HimanshuSinha-msft
19,381
Reputation points Microsoft Employee
0 answers
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 7.0 failed 4 times, most recent failure: java.lang.NoClassDefFoundError: Could not initialize class
Hi, I am getting this error despite defining the class. When I execute the notebook first time it works fine but when I execute the same notebook without code change it started throwing this error. As per the error class not defined but trust me class…
asked 2020-06-23T23:20:13.487+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(22.400000000000002, 49%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERK%3C/text%3E%3C/svg%3E)
Rajaniesh Kaushikk
476
Reputation points
commented 2020-07-02T22:39:19.027+00:00
KranthiPakala-MSFT
46,442
Reputation points Microsoft Employee
0 answers
data bricks scala : data frame column endoing from UTF 8 to windows 1252
HI I am working with data bricks where i have the data in parque and i am generating smaller files out of it , i have a column in this which is string and it has different characters and i have to encode this string value to windows 1252 or windows…
asked 2020-06-22T20:29:26.43+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(150.4, 88%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EM%3C/text%3E%3C/svg%3E)
ManojMathe
1
Reputation point
commented 2020-06-25T22:21:25.563+00:00
HimanshuSinha-msft
19,381
Reputation points Microsoft Employee
1 answer
Third party Python package installed on Databricks cluster gives different results than other Python stacks
We get a Python package developed by a third party. The package implements a standard mathematical model, no machine learning, no randomization. The model turned out to return incorrect results when installed on a Databricks cluster. We tried different…
asked 2020-06-18T07:56:40.197+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(118.4, 82%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHG%3C/text%3E%3C/svg%3E)
Hans Geurtsen
1
Reputation point
commented 2020-06-23T12:14:01.153+00:00
Hans Geurtsen
1
Reputation point
1 answer
Databricks Notebook Activity parameter problem
I feel this is a bug but not sure if it is with ADF or Databricks. I am running a notebook using ADF notebook activity. My notebook has a widget for which I pass the value from ADF. As I need to manually enter the parameter name while configuring…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(25.6, 95%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ET%3C/text%3E%3C/svg%3E)