Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,037 questions
7 answers
Spark U/I is not loading for recently "terminated" job clustsers
I've been having trouble with a feature in Azure Databricks. The Spark U/I will not be shown for job clusters that have recently completed ("terminated"). There is a link to the Spark U/I in the databricks portal. But when you click the…
asked 2021-04-07T20:49:06.167+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(70.4, 78%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDB%3C/text%3E%3C/svg%3E)
David Beavon
976
Reputation points
edited the question 2023-02-09T05:48:17.95+00:00
PRADEEPCHEEKATLA-MSFT
83,886
Reputation points Microsoft Employee
6 answers
I am not able to start my new and first databricks cluster
I have upgraded my plan, and also tried to change the region from East US to East US 2 , but even then my cluster is not starting error 1: subscription issue error 2 : Message Cluster terminated.Reason: Cloud provider launch failure A cloud…
asked 2022-05-28T20:16:57.883+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(67.2, 19%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ED%3C/text%3E%3C/svg%3E)
dev_pro
6
Reputation points
answered 2022-12-20T19:57:57.913+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(108.80000000000001, 1%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJB%3C/text%3E%3C/svg%3E)
Juggling Boy
6
Reputation points
6 answers
One of the answers was accepted by the question author.
Remove characters in from values pyspark
Hi Expert, How to remove characters from column values pyspark sql I.e gffg546, gfg6544
asked 2022-08-18T18:32:32.387+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(140.8, 75%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESR%3C/text%3E%3C/svg%3E)
Shambhu Rai
1,406
Reputation points
commented 2022-08-18T21:11:00.163+00:00
Dillon Silzer
54,931
Reputation points
6 answers
Databricks-Connect also return module not found for multiple python files job.
Currently I'm connecting to databricks with local VS Code via databricks-connect. But my submmission all comes with error of module not found, which means the code in other python files not found. I tried: Move code into the folder with main.py …
asked 2020-07-09T23:26:37.607+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 62%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDY%3C/text%3E%3C/svg%3E)
Dong Yuan
1
Reputation point
answered 2021-08-20T10:48:50.943+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(0, 4%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMU%3C/text%3E%3C/svg%3E)
Michał Urban
1
Reputation point
5 answers
How do I connect to Databricks with .NET and run delta table queries
Is there a way to connect to Databricks and run Delta table queries from .NET? I've seen a few approaches using ODBC/JDBC but is there some other which excludes these types of connection? Maybe managed identity? If applicable, where can I find some…
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,037 questions
C#
C#
An object-oriented and type-safe programming language that has its roots in the C family of languages and includes support for component-oriented programming.
10,542 questions
asked 2021-11-24T09:41:17.26+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(188.79999999999998, 95%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMK%3C/text%3E%3C/svg%3E)
Martin Kostadinov
36
Reputation points
answered 2023-04-24T15:37:43.24+00:00
Konstantinos Passadis
17,381
Reputation points MVP
5 answers
One of the answers was accepted by the question author.
Pyodbc install error in Azure Databricks: File not found error
Running the code below in Azure Databricks %sh curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add - curl https://packages.microsoft.com/config/ubuntu/16.04/prod.list > /etc/apt/sources.list.d/mssql-release.list apt-get…
asked 2021-06-16T23:25:50.96+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(16, 31%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECJ%3C/text%3E%3C/svg%3E)
Chung, Joshua
151
Reputation points
accepted 2021-06-17T15:58:23.647+00:00
Chung, Joshua
151
Reputation points
5 answers
One of the answers was accepted by the question author.
Adding datetime as dynamic content to a query
This is my query SELECT Id, ... FROM Task WHERE LastModifiedDate > @{activity('Lookup_SF_Task').output.firstRow.LastExtractRecordDate} This is how LastExtractRecordDate is being pulled by a Look Up module. This is the error…
asked 2021-04-09T13:02:01.443+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(108.80000000000001, 94%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBC%3C/text%3E%3C/svg%3E)
Bakar, Cem
281
Reputation points
commented 2021-04-12T16:11:44.407+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(172.8, 70%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
MartinJaffer-MSFT
26,046
Reputation points
5 answers
One of the answers was accepted by the question author.
Error when mounting Azure Data Lake Storage Gen1 account
I'm using the code below to mount gen1 account on azure databricks configs = {"fs.adl.oauth2.access.token.provider.type": "ClientCredential", "fs.adl.oauth2.client.id": <service-principal-appid>, …
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(54.400000000000006, 96%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
4 answers
how to resolve this error while getting data from databricks from power bi.Not able to load table in power bi
DataSource.Error: ODBC: ERROR [HY000] [Microsoft][DSI] (20039) Cannot store ""."".""."REMARKS" value in temporary table without truncation. (Column metadata implied a maximum of 512 bytes, while provided value is…
asked 2023-10-23T18:45:35.15+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(208, 11%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPK%3C/text%3E%3C/svg%3E)
Pratibha Khare
20
Reputation points
answered 2024-05-28T15:31:06.68+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(262.40000000000003, 67%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EM%3C/text%3E%3C/svg%3E)
Mike
0
Reputation points
4 answers
One of the answers was accepted by the question author.
What is the difference between Databrick prepay and Databrick reservation in Azure ?
Hello, We are just considering ways to reduce Databrick cost in Azure other than buying RI for VMs behind Databrick clusters. What is the difference between Databrick prepay and Databrick reservation in Azure It seems Databrick reservation is named as…
asked 2024-05-23T09:26:44.82+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(44.800000000000004, 76%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAK%3C/text%3E%3C/svg%3E)
Anil Kumar
265
Reputation points
accepted 2024-05-27T05:32:31.28+00:00
Anil Kumar
265
Reputation points
4 answers
Databricks Cluster Size
Hi, We have a below use case. We are developing ML Development using azure data bricks cluster service's. Our data bricks will receive 20 GB size of dataset records and we are running our logic/algorithms against this dataset's. We have to run every…
asked 2024-04-16T14:55:11.34+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(265.6, 60%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJV%3C/text%3E%3C/svg%3E)
james vasanth
0
Reputation points
commented 2024-04-18T09:05:04.9833333+00:00
PRADEEPCHEEKATLA-MSFT
83,886
Reputation points Microsoft Employee
4 answers
One of the answers was accepted by the question author.
Unable to create a Metastore in Azure Databricks
Hello awesome people! We are trying to implment Azure data bricks for our Proof of concept work, which involves the following. Gen 2 storage account with Hierarchical namespace enabled. Azure data bricks connector with the "Storage blob data…
asked 2023-08-11T14:37:50.3133333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(179.20000000000002, 3%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAJ%3C/text%3E%3C/svg%3E)
Ajinkya Joshi
20
Reputation points
edited an answer 2024-03-06T16:51:30.86+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(252.8, 79%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGJ%3C/text%3E%3C/svg%3E)
Geraldo, Jude
0
Reputation points
4 answers
Allow only a specific Azure Databricks instance to connect to keyvault
I want to connect to Azure Keyvault using "Azure Key Vault-backed scopes" in Databricks. Now on the Keyvault side, the way we allow an application to connect is via: RBAC - We are using the RBAC permission model. Add an identity in RBAC and…
asked 2022-01-11T01:21:28.393+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(147.20000000000002, 87%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENS%3C/text%3E%3C/svg%3E)
Nidhi Singh
6
Reputation points
commented 2023-08-03T08:48:24.2866667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(300.8, 45%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJK%3C/text%3E%3C/svg%3E)
Janne Kujanpää
206
Reputation points
4 answers
JDBC\ODBC connection to Azure Dataricks fails
Hi, I referred the following links in net and tried all, but still I am not connecting to Azure Databricks database https://docs.databricks.com/integrations/bi/power-bi.html https://docs.databricks.com/integrations/bi/jdbc-odbc-bi.html …
asked 2020-10-19T12:22:43.827+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(211.20000000000002, 40%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ER%3C/text%3E%3C/svg%3E)
Rajesh
36
Reputation points
commented 2023-07-04T11:29:50.2266667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(249.60000000000002, 96%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EZE%3C/text%3E%3C/svg%3E)
Zaid El Hassani
0
Reputation points
4 answers
Using Microsoft Azure Free Trial for Databricks
I want to try out Databricks using my free trial. I am able to create a workspace, but when I try to create and run a cluster within it I get an error saying I have exceeded my core quota for the free trial. Is there a way I can try out Databricks…
asked 2020-09-10T16:02:49.623+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(211.20000000000002, 81%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ED%3C/text%3E%3C/svg%3E)
dfhgffg
1
Reputation point
edited the question 2023-02-07T21:52:07.5866667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 9%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKM%3C/text%3E%3C/svg%3E)
KranthiPakala-MSFT
46,437
Reputation points Microsoft Employee
4 answers
Why SQL Server (and when) instead of Data Bricks or Data Lake
Firstly Apologies! if my question is not placed on the correct forum, I am really unsure, in which forum I should raise it........... ~ I am a SQL developer (and I do a little bit of Administration too) and currently trying to learn topics like Data…
asked 2022-11-29T11:29:09.547+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(76.8, 35%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAM%3C/text%3E%3C/svg%3E)
Avik Mukherjee
1
Reputation point
answered 2022-11-30T02:05:14.083+00:00
PandaPan-MSFT
1,906
Reputation points
4 answers
One of the answers was accepted by the question author.
Github Integration Failure code 404
I have followed the instructions to integrate Databricks with GitHub, found here: https://docs.databricks.com/notebooks/github-version-control.html However, I continue to get the following failure error upon trying to sync my notebooks: "Error…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(60.8, 28.999999999999996%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKM%3C/text%3E%3C/svg%3E)
4 answers
Databricks vnet peering
Environment: Single Azure subscription One resource group rgA, containing a vnet vnetA and some other resources I have contributor rights to this resource group. Done: I created a Databricks workspace myDatabricks. I…
asked 2021-11-01T17:27:40.277+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(28.799999999999997, 34%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJV%3C/text%3E%3C/svg%3E)
Johan Van Noten
1
Reputation point
answered 2021-11-03T17:22:36.237+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 59%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECM%3C/text%3E%3C/svg%3E)
ChaitanyaNaykodi-MSFT
24,231
Reputation points Microsoft Employee
4 answers
One of the answers was accepted by the question author.
Azure Databricks Synapse Connectivity
We are trying to use PolyBase in Azure Data Factory to copy the Delta lake table to Synapse. Using a simple Copy Activity in Azure Data Factory, our linked Services connections from Delta lake and Synapse show connection is successful, yet the copy step…
asked 2021-03-29T20:30:55.797+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 41%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESM%3C/text%3E%3C/svg%3E)
Sahar Mostafa
26
Reputation points
answered 2021-05-26T16:30:02.147+00:00
brajesh jaishwal
26
Reputation points
4 answers
com.databricks.sql.io.FileReadException: Error while reading file wasbs:REDACTED_LOCAL_PART@****.blob.core.windows.net/
Hi, I am getting the following error message: com.databricks.sql.io.FileReadException: Error while reading file…
asked 2021-01-29T20:32:13.813+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(89.60000000000001, 23%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMK%3C/text%3E%3C/svg%3E)
Mayuri Kadam
81
Reputation points Microsoft Employee
answered 2021-04-13T12:00:41.21+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(198.4, 89%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EEM%3C/text%3E%3C/svg%3E)
Eugen Mirosch
1
Reputation point