Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,910 questions
1 answer
execution plan in databricks
How to check execution plan in data bricks select * from cte
asked 2024-04-15T11:33:12.4566667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 13%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVS%3C/text%3E%3C/svg%3E)
Vineet S
145
Reputation points
commented 2024-04-19T23:10:18.5666667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(86.4, 36%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBM%3C/text%3E%3C/svg%3E)
BhargavaGunnam-MSFT
25,881
Reputation points Microsoft Employee
1 answer
coalesce and broadcast join
HI, what exactly happen between coalesce and broadcast join in backend on databricks level
asked 2024-04-19T09:04:04.1866667+00:00
Vineet S
145
Reputation points
answered 2024-04-19T12:30:35.95+00:00
Amira Bedhiafi
14,881
Reputation points
1 answer
How to recover a accidently deleted databricks instance in a free trial account
How to recover a accidently deleted databricks instance in a free trial account.
asked 2024-04-19T10:01:26.8066667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(128, 99%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVJ%3C/text%3E%3C/svg%3E)
Vinee Jain
0
Reputation points
answered 2024-04-19T11:32:10.55+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 8%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EST%3C/text%3E%3C/svg%3E)
Smaran Thoomu
9,045
Reputation points Microsoft Vendor
1 answer
partition in db
HI , what hppend in databricks backend when partion is applied to table level
asked 2024-04-19T09:05:58.03+00:00
Vineet S
145
Reputation points
answered 2024-04-19T09:28:39.7533333+00:00
Vinodh247-1375
11,031
Reputation points
1 answer
How can I delete my Azure Databricks workspace when the workspace resource has already been removed?
As the title says I have an Azure Databricks Workspace where I have deleted the Workspace resource in Azure. However, the workspace still appears in the Account Console. From here I can log into the workspace but not create clusters etc. When I click on…
asked 2024-04-18T11:20:48.4533333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(252.8, 10%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKR%3C/text%3E%3C/svg%3E)
Kim Reinhart Jensen (KJEN)
0
Reputation points
commented 2024-04-19T09:26:35.24+00:00
Kim Reinhart Jensen (KJEN)
0
Reputation points
1 answer
Azure Databricks fail to install Geospark libraries from Maven
Hi Team , I am attempting to add below two geospark Maven libraries to my Azure Databricks interactive cluster with Runtime Version 14.3 LTS . However , I am getting below error Library installation attempted on the driver node of cluster…
asked 2024-04-15T06:24:17.8033333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(96, 57.99999999999999%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAS%3C/text%3E%3C/svg%3E)
Anuj, Singh (Cognizant)
0
Reputation points
commented 2024-04-19T08:19:03.06+00:00
PRADEEPCHEEKATLA-MSFT
76,586
Reputation points Microsoft Employee
0 answers
FileAlreadyExistsException: Failed to rename temp file dbfs:/mnt/delta_checkpoints/sources/0/rocksdb/__tmp_path_dir/.2.zip.52d0723f-b803-4a8a-9533-9d6e67813641.tmp to dbfs:/mnt/delta_checkpoints/sources/0/rocksdb/2.zip because file exists
I have built a streaming pipeline with spark autoloader. Source Folder is a azure blob container. We encountered a rare issue (could not replicate it). Below is the exception Message: org.apache.hadoop.fs.FileAlreadyExistsException: Failed to…
asked 2022-03-23T22:43:18.8+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(294.40000000000003, 57.00000000000001%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBS%3C/text%3E%3C/svg%3E)
Balasubramanian Singaravelu
6
Reputation points
commented 2024-04-19T08:04:21.1366667+00:00
PRADEEPCHEEKATLA-MSFT
76,586
Reputation points Microsoft Employee
2 answers
Failing to connect to metastore when using dbx to launch ephimeral cluster in databricks
Dear all, I am using dbx to deploy and launch jobs on ephemeral clusters on databricks. I have initialized the the cicd-sample-project and connected to a fresh empty Databricks Free trial environment and everything works. But when I try to do…
asked 2022-12-13T12:39:33.237+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(96, 80%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EEM%3C/text%3E%3C/svg%3E)
Enrico Mosca
6
Reputation points
answered 2024-04-19T07:07:13.3133333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(246.4, 53%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
Alexander
0
Reputation points
1 answer
Azure Delta Lake to Snowflake
Hi Team, I am creating Delta lake in Azure data lake from ADF using Dataflow Sink - inline dataset as Delta and also through Databricks. Have created External Table in Databricks which is pointing to Mounted Azure datalake location. Now, I want to load…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(83.2, 24%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EV%3C/text%3E%3C/svg%3E)
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 61%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EP%3C/text%3E%3C/svg%3E)
0 answers
Custom text single label classification - Model API Consumption within Databricks
Hello together, i trained a model within Azure Language Studio , Custom text single label classification - and i want to consume Model API within Databricks Notebook. I get always below given error, kindly asking for your help. Thanks. Error: HTTP…
asked 2024-04-18T11:00:16.9066667+00:00
Aziz Öztürk
20
Reputation points
commented 2024-04-18T11:54:11.46+00:00
PRADEEPCHEEKATLA-MSFT
76,586
Reputation points Microsoft Employee
4 answers
Databricks Cluster Size
Hi, We have a below use case. We are developing ML Development using azure data bricks cluster service's. Our data bricks will receive 20 GB size of dataset records and we are running our logic/algorithms against this dataset's. We have to run every…
asked 2024-04-16T14:55:11.34+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(265.6, 60%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJV%3C/text%3E%3C/svg%3E)
james vasanth
0
Reputation points
commented 2024-04-18T09:05:04.9833333+00:00
PRADEEPCHEEKATLA-MSFT
76,586
Reputation points Microsoft Employee
1 answer
How to calculate the price of a databricks job with Photon engine?
It's trivial to calculate a databricks job cost using Azure pricing calculator. You add up Azure VM price and Azure Databricks DBU consumption price and you would get your answer. However when it comes to Databricks with Photon runtime, the pricing is…
asked 2024-04-05T14:53:12.39+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(259.20000000000005, 65%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJZ%3C/text%3E%3C/svg%3E)
Jiapeng Zhang
0
Reputation points
commented 2024-04-17T07:50:19.56+00:00
PRADEEPCHEEKATLA-MSFT
76,586
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
ADF pipeline to read the data from UC table to adls gen2 account
Hello Team, We have a requirement to create Azure Datafactory pipeline to read the data from UC table, access on the table is provided ( to Azure Datafactory Managed Identity) and copy the data into adls gen2. Is there a way or article to implement this?…
asked 2024-04-11T19:05:05.9733333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 24%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAG%3C/text%3E%3C/svg%3E)
Ashwini Gaikwad
65
Reputation points
accepted 2024-04-15T07:35:40.09+00:00
Ashwini Gaikwad
65
Reputation points
0 answers
Enabling Azure PIM Disables user within DataBricks
We have successfully stood up Azure Databricks in our tenant. We are leveraging SCIM and User Provisioning to grant our users SSO access into DataBricks. We are trying to layer in an additional layer of security to meet our current security standards by…
asked 2023-09-01T14:55:42.55+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(211.20000000000002, 66%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
Smileyville
1
Reputation point
edited a comment 2024-04-15T01:04:28.3066667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(105.60000000000001, 32%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EC%3C/text%3E%3C/svg%3E)
Chris
0
Reputation points
2 answers
How to Fix Error Configuring VPC Peering in Azure Databricks? Failed to add virtual network peering 'Peering' to 'workers-vnet'. Error: The client "" with object ID "" has permission to perform action
I'm facing an issue while attempting to configure VPC peering in Azure Databricks. When trying to establish VPC peering between the Azure Databricks workspace's VNET ("workers-vnet") and an external network, I encountered the following error…
asked 2024-04-10T17:31:13.91+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(163.2, 26%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EID%3C/text%3E%3C/svg%3E)
Ivan David Perez Moreno
0
Reputation points
commented 2024-04-12T07:02:36.9066667+00:00
PRADEEPCHEEKATLA-MSFT
76,586
Reputation points Microsoft Employee
0 answers
Can we customize the index url to fetch Python package from a protected PyPi source
Recently Databricks provided in public preview the possibility to add library in the compute policy. We will need to install a library located in a protected PyPi repo from Azure DevOps: pkgs.dev.azure.com/{further_path}/pypi/packages. We used to have…
asked 2024-03-27T10:28:37.7933333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(140.8, 9%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESA%3C/text%3E%3C/svg%3E)
Sypula, Aleksandra
0
Reputation points
commented 2024-04-12T06:22:22.8666667+00:00
PRADEEPCHEEKATLA-MSFT
76,586
Reputation points Microsoft Employee
1 answer
Sync the tables from one azure databricks workspace to other databricks workspace or to adls gen2
Hello Team, We have two UC enabled databricks workspace. And we have to sync tables created in one Azure databricks workspace to other databricks workspace using PAT/any other reliable way or to adls gen2 account. Request you to let me know is there a…
asked 2024-04-11T18:50:09.1566667+00:00
Ashwini Gaikwad
65
Reputation points
answered 2024-04-11T21:16:40.9833333+00:00
BhargavaGunnam-MSFT
25,881
Reputation points Microsoft Employee
1 answer
How to add SSL certificate for using it on Databricks cluster with PySpark
Hi, We would like to use functions written in PySpark for calling an external service that requires SSL certificate on the cluster. Currently we are using an init script similar to explained in the documentation -…
asked 2024-04-03T08:20:22.1+00:00
Sypula, Aleksandra
0
Reputation points
commented 2024-04-11T04:23:37.9833333+00:00
PRADEEPCHEEKATLA-MSFT
76,586
Reputation points Microsoft Employee
1 answer
How can you use Databricks Vector Search with images?
I have been working with Databricks Vector Search and feel comfortable using it on any kind of textual data. However I am running into problems with using it for image searching, and there are no examples available in the documentation to provide more…
asked 2024-03-21T22:17:02.7866667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(246.4, 93%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EI%3C/text%3E%3C/svg%3E)
Isabel
0
Reputation points
commented 2024-04-11T04:20:46.4466667+00:00
PRADEEPCHEEKATLA-MSFT
76,586
Reputation points Microsoft Employee
1 answer
¿Como puedo solucionar un problema con los cargos de unos recursos que no se elminaron y me generaron una deuda?
En diciembre, comencé a aprender databricks y segui un tutorial de azure y al finalizar eliminé los servicios para que no me cobraran y ya me llegó un cobro de 3 pesos por lo que usé el mes, total segui con los tutoriales y un dia comencé en la mañana…
asked 2024-04-07T03:35:33.39+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(92.8, 69%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECR%3C/text%3E%3C/svg%3E)
CESAR RUIZ FLORES
0
Reputation points
commented 2024-04-10T04:48:07.67+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(220.8, 4%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESM%3C/text%3E%3C/svg%3E)
ShaktiSingh-MSFT
13,271
Reputation points Microsoft Employee