Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,917 questions
0 answers
Raising the limit in Azure Databricks identities
Context: The Azure Databricks documentation states that "You can have a maximum of 10,000 combined users and service principals and 5,000 groups in an account. Each workspace can have a maximum of 10,000 combined users and service principals and…
asked 2024-04-25T10:41:16.95+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(156.8, 28.999999999999996%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKS%3C/text%3E%3C/svg%3E)
Kim Stig Hansen
0
Reputation points
asked 2024-04-25T10:41:16.95+00:00
Kim Stig Hansen
0
Reputation points
1 answer
Sync the tables from one azure databricks workspace to other databricks workspace or to adls gen2
Hello Team, We have two UC enabled databricks workspace. And we have to sync tables created in one Azure databricks workspace to other databricks workspace using PAT/any other reliable way or to adls gen2 account. Request you to let me know is there a…
asked 2024-04-11T18:50:09.1566667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 24%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAG%3C/text%3E%3C/svg%3E)
Ashwini Gaikwad
65
Reputation points
commented 2024-04-25T10:38:55.3233333+00:00
Ashwini Gaikwad
65
Reputation points
1 answer
One of the answers was accepted by the question author.
How to use AWS Databricks note book in ADF..?
Hi Team, I want to create Linked service in ADF with AWS Databricks notebook. Can you guys help me with list of steps that i need follow. Regards, Naveen.
asked 2023-10-30T03:44:41.43+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(316.8, 77%, 40%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENK%3C/text%3E%3C/svg%3E)
Naveen Kumar Rachapalle (EXT/DIGITAL)
25
Reputation points
commented 2024-04-25T10:09:37.6533333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(35.2, 66%, 13%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESA%3C/text%3E%3C/svg%3E)
Syed Ahamed S
0
Reputation points
1 answer
Custom text single label classification - Model API Consumption within Databricks
Hello together, i trained a model within Azure Language Studio , Custom text single label classification - and i want to consume Model API within Databricks Notebook. I get always below given error, kindly asking for your help. Thanks. Error: HTTP…
asked 2024-04-18T11:00:16.9066667+00:00
Aziz Öztürk
20
Reputation points
commented 2024-04-25T05:47:24.94+00:00
PRADEEPCHEEKATLA-MSFT
76,921
Reputation points Microsoft Employee
1 answer
How can I delete my Azure Databricks workspace when the workspace resource has already been removed?
As the title says I have an Azure Databricks Workspace where I have deleted the Workspace resource in Azure. However, the workspace still appears in the Account Console. From here I can log into the workspace but not create clusters etc. When I click on…
asked 2024-04-18T11:20:48.4533333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(252.8, 10%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKR%3C/text%3E%3C/svg%3E)
Kim Reinhart Jensen (KJEN)
0
Reputation points
commented 2024-04-25T05:46:32.7666667+00:00
PRADEEPCHEEKATLA-MSFT
76,921
Reputation points Microsoft Employee
1 answer
How to fix timeout error when clreating compute cluster on azure databricks.
Azure error message: [id: InstanceId(e8b82690e17a4624a152afb20dce5339), status: INSTANCE_LAUNCHING, workerEnvId:WorkerEnvId(workerenv-3367469989580530), lastStatusChangeTime: 1713836219115, groupIdOpt Some(0),requestIdOpt…
asked 2024-04-23T02:31:21.1133333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(22.400000000000002, 21%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGG%3C/text%3E%3C/svg%3E)
GEOFREY GETUBA
0
Reputation points
commented 2024-04-25T05:39:28.3766667+00:00
PRADEEPCHEEKATLA-MSFT
76,921
Reputation points Microsoft Employee
1 answer
How create a unity catalogue for existing Azure Databricks workspaces with all permissions as identity governance administrator?
Trying to create a unity catalogue for easy flow of machine learning solution on Databricks which has 3 separate workspaces - development, staging and production.
asked 2024-04-24T03:22:24.2566667+00:00
Mohd Zubair Humza
0
Reputation points
commented 2024-04-25T05:37:09.69+00:00
PRADEEPCHEEKATLA-MSFT
76,921
Reputation points Microsoft Employee
1 answer
An error occurred when using odbc and managed identity to link azure databricks in the. net core
Environmental description: The app service has already been configured with managed identity, and the relevant permissions for this managed identity have been configured in Azure databricks. When the app service uses code to access Azure databricks, the…
asked 2024-04-23T05:55:31.6166667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(316.8, 79%, 40%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EZZ%3C/text%3E%3C/svg%3E)
ZhiHong Zhang (BEYONDSOFT CONSULTING INC)
0
Reputation points
commented 2024-04-25T05:18:33.11+00:00
PRADEEPCHEEKATLA-MSFT
76,921
Reputation points Microsoft Employee
1 answer
Databricks jobs are failing with the insufficient privileges error
When running the jobs in databrick user is getting the insufficient privilege error, but user do have all the privileges like select, read_metadata, modify, create and owner as well on the respective database.
asked 2024-04-24T19:28:13.1033333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(51.2, 24%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECB%3C/text%3E%3C/svg%3E)
Chethana B S
0
Reputation points
answered 2024-04-25T03:54:21.4333333+00:00
PRADEEPCHEEKATLA-MSFT
76,921
Reputation points Microsoft Employee
2 answers
execution plan in databricks
How to check execution plan in data bricks select * from cte
2 answers
coalesce and broadcast join
HI, what exactly happen between coalesce and broadcast join in backend on databricks level
3 answers
partition in db
HI , what hppend in databricks backend when partion is applied to table level
asked 2024-04-19T09:05:58.03+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 13%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVS%3C/text%3E%3C/svg%3E)
Vineet S
145
Reputation points
commented 2024-04-24T13:17:39.69+00:00
ShaikMaheer-MSFT
37,896
Reputation points Microsoft Employee
2 answers
One of the answers was accepted by the question author.
How to ingest CDC data from SQL DATA SYNC to Event hub?
Hi we have a scenario to implement real-time CDC from SQL DATA SYNC to Azure SQL DB. Which tools are good for near real-time processes and prices as well. By using data bricks we can merge stream data into delta lake. But How to ingest real-time…
asked 2020-11-10T12:28:43.067+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(16, 54%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ETG%3C/text%3E%3C/svg%3E)
THIMMAIAH GARI,PRASHANTH,,
201
Reputation points
edited an answer 2024-04-24T06:51:50.8266667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(195.2, 91%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESB%3C/text%3E%3C/svg%3E)
Shane Blake
0
Reputation points
1 answer
How to write to datalakegen2 storage using databricks in delta format when connected using SAS tocken
I converted the data from parquet form to data format. Now I want to write the data to blob storage - datalakegen2. But facing below error while writing. I useed the below command to write my data: output_path =…
asked 2024-04-09T23:29:12.6666667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(217.60000000000002, 55.00000000000001%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESS%3C/text%3E%3C/svg%3E)
Sai Sunny Kothuri
0
Reputation points
commented 2024-04-24T05:00:35.0166667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(118.4, 98%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKM%3C/text%3E%3C/svg%3E)
KarishmaTiwari-MSFT
18,367
Reputation points Microsoft Employee
0 answers
DatabricksSQL Logs and correlate with query history
Hi everyone, I'm currently working on capturing logging information about query executions and data downloads within a Databricks workspace. Here's a summary of my current setup and the issue I'm facing: Diagnostic Settings in Azure Databricks: I have…
asked 2024-04-22T11:09:19.4+00:00
Julio Avellaneda
0
Reputation points
commented 2024-04-24T02:51:51.83+00:00
PRADEEPCHEEKATLA-MSFT
76,921
Reputation points Microsoft Employee
0 answers
SAP latency data
Hi Expert, how to we can load the data from modified data in updated or insert fields in databricks using ADF or data bricks on trigger level instead of loading multiple times example: table updated or inserted with new records how table change and…
2 answers
LivyHttpRequestFailure: Something went wrong while processing your request. Please try again later. HTTP status code: 500. Trace ID: d2425fce-9179-49eb-829c-a7b2a0d963ed.
Hi Everyone, I am getting the Below Error Code while running my cells in Synapse Notebook: > LivyHttpRequestFailure: Something went wrong while processing your request. Please try again later. HTTP status code: 500. Trace ID:…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(44.800000000000004, 15%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ED%3C/text%3E%3C/svg%3E)
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(86.4, 93%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDS%3C/text%3E%3C/svg%3E)
1 answer
Azure Databricks fail to install Geospark libraries from Maven
Hi Team , I am attempting to add below two geospark Maven libraries to my Azure Databricks interactive cluster with Runtime Version 14.3 LTS . However , I am getting below error Library installation attempted on the driver node of cluster…
asked 2024-04-15T06:24:17.8033333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(96, 57.99999999999999%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAS%3C/text%3E%3C/svg%3E)
Anuj, Singh (Cognizant)
5
Reputation points
commented 2024-04-23T03:38:08.5466667+00:00
PRADEEPCHEEKATLA-MSFT
76,921
Reputation points Microsoft Employee
1 answer
How to recover a accidently deleted databricks instance in a free trial account
How to recover a accidently deleted databricks instance in a free trial account.
asked 2024-04-19T10:01:26.8066667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(128, 99%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVJ%3C/text%3E%3C/svg%3E)
Vinee Jain
0
Reputation points
commented 2024-04-22T03:54:20.6066667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 8%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EST%3C/text%3E%3C/svg%3E)
Smaran Thoomu
9,525
Reputation points Microsoft Vendor
1 answer
One of the answers was accepted by the question author.
Azure Delta Lake to Snowflake
Hi Team, I am creating Delta lake in Azure data lake from ADF using Dataflow Sink - inline dataset as Delta and also through Databricks. Have created External Table in Databricks which is pointing to Mounted Azure datalake location. Now, I want to load…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(83.2, 24%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EV%3C/text%3E%3C/svg%3E)