Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,091 questions
0 answers
Loading data from .csv
Today I am unable to load data from a .csv into Dynamics365 Sales. Earlier in the day, I deleted all of our records with the hope of importing the data again with a new mapping system. I have tried to upload throughout the day. Sometimes, nothing…
asked 2020-10-23T17:53:56.787+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(227.2, 71%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPA%3C/text%3E%3C/svg%3E)
Peter Abraldes
1
Reputation point
commented 2020-11-02T22:32:50.847+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 26%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HimanshuSinha-msft
19,461
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Getting error while using databricks command in ubuntu Vm even after installing databricks-cli
Hi , I am working on one requirement of monitoring databricks with azure monitor. and i m using below document for this .. https://github.com/mspnp/spark-monitoring in one my ubuntu Vm i have completed all pre-requisites .. i also installed…
asked 2020-10-30T10:02:21.333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(35.2, 28.000000000000004%, 13%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERB%3C/text%3E%3C/svg%3E)
Rohit Boddu
466
Reputation points
accepted 2020-11-02T05:27:06.867+00:00
Rohit Boddu
466
Reputation points
0 answers
How to integrate Databricks with PowerBI
Hi, I want to integrate Azure Databricks with PowerBI.. Below is my requirement .. I want to generate Power Bi Reports out of Adls using hive tables .. How can i achieve this ? But i m getting below error attached in screenshot while…
asked 2020-10-30T10:17:09.853+00:00
Rohit Boddu
466
Reputation points
commented 2020-11-02T05:00:29.12+00:00
PRADEEPCHEEKATLA-MSFT
85,981
Reputation points Microsoft Employee
1 answer
I would pay cash money for a graphical UI on the portal/dashboard.
Once you get to 200+ resources, the dashboard is a complete disaster. Graphical UI! Make it look like a Windows desktop for all I care.
asked 2020-10-30T15:36:27.34+00:00
Black, Michael
1
Reputation point
answered 2020-10-30T21:25:58.077+00:00
JamesTran-MSFT
36,541
Reputation points Microsoft Employee
3 answers
One of the answers was accepted by the question author.
Power BI custom columns
Hi everyone, I need to add a custom column in Power BI Edit Query, where the custom column is calculated from another table. Please help. Can this be done? Looking forward to your kind feedback and assistance. Regards Thabo
asked 2020-07-07T07:57:50.727+00:00
Thabo Mofokeng
21
Reputation points
answered 2020-10-30T05:21:27.51+00:00
Rachit Malik
1
Reputation point
1 answer
One of the answers was accepted by the question author.
Azure Databricks queries for monitoring using log analytics
Hi, I have premium version of azure databricks and i have enabled monitoring as well in this .. so i have sent its all data to log analytics workspace .. now i want to query on databricks .. can u provide me with written queries so i can use them in my…
asked 2020-10-20T06:30:51.913+00:00
Rohit Boddu
466
Reputation points
accepted 2020-10-30T03:42:23.637+00:00
Rohit Boddu
466
Reputation points
1 answer
One of the answers was accepted by the question author.
Locking down a port on Azure Databricks (managed resoruce)
I am trying to lock down a high risk port on a managed azure databricks workspace. My problem is the exact opposite of the request on the question "Opening up port 443 on Azure Databricks (managed resource)". Instead of wanting to open up…
asked 2020-10-20T00:21:22.12+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(54.400000000000006, 43%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDW%3C/text%3E%3C/svg%3E)
Daniel Widdis
21
Reputation points
commented 2020-10-28T03:31:52.943+00:00
PRADEEPCHEEKATLA-MSFT
85,981
Reputation points Microsoft Employee
2 answers
One of the answers was accepted by the question author.
Read files from Azure File share using databricks
How can I read files stored in Azure file share using databricks. I am able to access container blob storage, but not the files in file share.
asked 2020-10-21T08:52:47.307+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(48, 91%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJ%3C/text%3E%3C/svg%3E)
Jeeva
161
Reputation points
commented 2020-10-28T03:28:44.43+00:00
PRADEEPCHEEKATLA-MSFT
85,981
Reputation points Microsoft Employee
asked 2020-10-25T16:16:37.29+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(179.20000000000002, 97%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBH%3C/text%3E%3C/svg%3E)
BATRAH HAMZA
1
Reputation point
answered 2020-10-26T14:53:48.253+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(214.4, 45%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGM%3C/text%3E%3C/svg%3E)
GiftA-MSFT
11,161
Reputation points
0 answers
How to call websocket api and make a data set based on time?
I want to call websocket api from data anlytics or data factory which gives continous data from api. i want to read that data and based on time after every 5 minutes want to take out open high low and close values from that set. i can able to do it using…
asked 2020-10-11T07:36:44.293+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(294.40000000000003, 13%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAF%3C/text%3E%3C/svg%3E)
ashish fugat
11
Reputation points
commented 2020-10-22T16:58:21.693+00:00
HimanshuSinha-msft
19,461
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Azure databricks workload type
I am calling a azure databricks cluster from a AI/ML pipeline. I want to understand it will get executed under which feature type: Data analytics or data engineering. I want to understand this to get my databricks usage cost calculation.
asked 2020-10-19T21:51:01.31+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 55.00000000000001%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESP%3C/text%3E%3C/svg%3E)
Sandhya Prusty
21
Reputation points
commented 2020-10-22T07:10:38.28+00:00
PRADEEPCHEEKATLA-MSFT
85,981
Reputation points Microsoft Employee
1 answer
Why we dont have diagnostic settings feature in Monitoring Section of "Azure Databricks of standard version"
Hi, I want to monitor Azure Databricks using Azure monitor .. I first want to send all its diagnostics settings to log analytics and then i can analyze from there. but there is no diagnostics setting feature Available for "Azure databricks Standard…
asked 2020-10-16T09:32:25.407+00:00
Rohit Boddu
466
Reputation points
commented 2020-10-21T11:15:44.667+00:00
PRADEEPCHEEKATLA-MSFT
85,981
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Azure Databricks table
Hi, One of my Datascientist by mistakenly delete data file working on Azure Databricks. Now How to restore deleted table from Azure Data Lake?
asked 2020-10-16T19:55:36.973+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(16, 46%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVK%3C/text%3E%3C/svg%3E)
Vijay Kumar
2,031
Reputation points
accepted 2020-10-20T16:05:50.507+00:00
Vijay Kumar
2,031
Reputation points
2 answers
How to maintain uniqueness in dataframe in databricks
When you load data into dataframes in databricks how can you make sure the rows in dataframes are not duplicated. In SQL you can handle by using unique constraint on the tables. how this can be handle in dataframes to ensure rows are not duplicated.
asked 2020-10-13T13:01:40.6+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(137.6, 0%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAG%3C/text%3E%3C/svg%3E)
Abhishek Gaikwad
191
Reputation points
commented 2020-10-15T18:31:35.803+00:00
HimanshuSinha-msft
19,461
Reputation points Microsoft Employee
2 answers
One of the answers was accepted by the question author.
Getting SSL certificate error while connecting to Synapse SQL pool from Databricks in Azure using JDBC
--------------------------------------------------------------------------- Py4JJavaError Traceback (most recent call last) <command-3031308706598466> in <module> 6 .option("tempDir",…
asked 2020-10-06T17:17:39.827+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(150.4, 91%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAV%3C/text%3E%3C/svg%3E)
Ajay Verma
21
Reputation points
commented 2020-10-15T05:11:29.787+00:00
PRADEEPCHEEKATLA-MSFT
85,981
Reputation points Microsoft Employee
1 answer
notebook-scoped lib failing to find the dependent cluster-scoped lib
Our app hit an issue with ModuleNotFoundError recently with Databricks Runtime 6.4 and it looks this is a databricks issue, so wanted to ask about that. The problem can happen as follows: Two libraries A.B and A.C; A.C depends on A.B Install A.B…
asked 2020-10-07T21:45:54.727+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(313.6, 41%, 40%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ETT%3C/text%3E%3C/svg%3E)
Tri Tran
1
Reputation point Microsoft Employee
answered 2020-10-14T21:35:03.173+00:00
HimanshuSinha-msft
19,461
Reputation points Microsoft Employee
4 answers
Unable to read csv file from cosmos gen1 through spark. Pandas read works.
This works, df = pd.read_csv('/dbfs/mnt/ajviswan/forest_efficiency/2020-04-26_2020-05-26.csv') sdf = spark.createDataFrame(df) sdf.head() But spark read does not work. df =…
asked 2020-10-08T15:35:40.867+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(195.2, 91%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAP%3C/text%3E%3C/svg%3E)
Ajay Prasadh Viswanathan
1
Reputation point Microsoft Employee
commented 2020-10-14T11:34:54.997+00:00
Ajay Prasadh Viswanathan
1
Reputation point Microsoft Employee
1 answer
Azure Databricks Access Controls in Programmatic way
Can someone please let me know is there any programmatic way to implement Access control on Azure Databricks on Workspace object, cluster, pool, job, and table etc.. Any powershell or API for the same ?
asked 2020-10-02T11:20:29.373+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(137.6, 61%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAV%3C/text%3E%3C/svg%3E)
Akash Verma
21
Reputation points
commented 2020-10-14T10:18:58.663+00:00
PRADEEPCHEEKATLA-MSFT
85,981
Reputation points Microsoft Employee
1 answer
Azure Databrick workspace linking to Azure Machine Learning workspace error unable to get Workspace.from_config()
We are in a POC of Azure Databricks and Azure Machine Learning integration so that we can advance our MLOps practice. We have been dealing the with workspace linking issue for the past few weeks and have hit a wall. We worked with Databricks and search…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(3.2, 33%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ETD%3C/text%3E%3C/svg%3E)
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(307.2, 83%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESS%3C/text%3E%3C/svg%3E)
1 answer
One of the answers was accepted by the question author.
Writing parquet file throws…An HTTP header that's mandatory for this request is not specified
I have two ADLSv2 storage accounts, both are hierarchical namespace enabled. In my Python Notebook, I'm reading a CSV file from one storage account and writing as parquet file in another storage, after some enrichment. I am getting below error when…
asked 2020-10-12T23:48:19.033+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 33%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPM%3C/text%3E%3C/svg%3E)
Porsche Me
131
Reputation points
accepted 2020-10-13T13:44:57.697+00:00
Porsche Me
131
Reputation points