Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,045 questions
1 answer
How to connect to Azure Active Directory data using LDAP server
We are trying to get Azure Active Directory data like AD Users, AD Groups into ADLS. There is one LDAP server configured. How to fetch data from AD into ADLS. Please let me know in detail. Thank you for your help.
asked 2024-07-04T10:27:37.02+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(156.8, 77%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ELN%3C/text%3E%3C/svg%3E)
Lakshmi Narayana Sarma Bhamidipati
25
Reputation points
commented 2024-07-04T14:52:08.52+00:00
Lakshmi Narayana Sarma Bhamidipati
25
Reputation points
2 answers
how to fetch data from Azure Active Directory(AD) by using either ADF or databricks
To fetch data from Azure Active Directory (AD) using either Azure Data Factory (ADF) or Azure Databricks, Pleae let me know in detail. thanks
asked 2024-07-04T10:41:44.7933333+00:00
Lakshmi Narayana Sarma Bhamidipati
25
Reputation points
edited a comment 2024-07-04T13:26:31.14+00:00
Lakshmi Narayana Sarma Bhamidipati
25
Reputation points
0 answers
Guidance on how to use Service Principal with Certificate to Authorize for EventHub Stream Read
I found this documentation https://github.com/Azure/azure-event-hubs-spark/blob/master/docs/use-aad-authentication-to-connect-eventhubs.md online on how to use service principal with certificate to use spark stream read from EventHubs, I want to do this…
asked 2024-07-01T21:27:20.97+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(67.2, 28.999999999999996%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EB%3C/text%3E%3C/svg%3E)
BEPV
0
Reputation points
commented 2024-07-04T09:41:49.15+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 8%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EST%3C/text%3E%3C/svg%3E)
Smaran Thoomu
12,090
Reputation points Microsoft Vendor
1 answer
How to assign identity to azure databricks
How to assign identity to azure databricks as identity option is not available in the portal as per the MS doc
asked 2024-07-03T16:44:53.5+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(230.39999999999998, 85%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGD%3C/text%3E%3C/svg%3E)
Ghosh Dastidar, Priyanka (Cognizant)
0
Reputation points
answered 2024-07-03T16:55:39.68+00:00
Manu Philip
17,186
Reputation points MVP
1 answer
One of the answers was accepted by the question author.
Downsizing subnets that are associated with a databricks workspace
Hi, Is it possible to downsize the private and public subnets that currently have nothing attached to it but are use by a databricks workspace? Would this require the databricks cluster to be redeployed? Are there any extra steps needed so the…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(140.8, 61%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
1 answer
Can we connect ADX to Databricks, without using App Registration.?
I am trying to Access ADX data into Databricks, but failing to configure connection between databricks and ADX. I don't have permissions to use App registration. Is there any way to do with managed identity or any other alternative.
asked 2024-06-25T12:25:47.8766667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(243.2, 10%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERA%3C/text%3E%3C/svg%3E)
Ram Ahuja, Roma
0
Reputation points
edited the question 2024-07-03T10:29:01.5066667+00:00
PRADEEPCHEEKATLA-MSFT
84,371
Reputation points Microsoft Employee
0 answers
While running SQL query in Azure Databricks workspace i.e. on SQL warehouse as well as on UC enabled shared cluster facing an SSL handshake error
Hello Team, We have UC enabled Azure databricks workspace, also the Public access and delta sharing is disabled on our workspace. So while running the below SQL query on SQL Warehouse as well as on UC enabled shared cluster, I am receiving an…
asked 2024-06-28T10:06:18.7566667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 24%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAG%3C/text%3E%3C/svg%3E)
Ashwini Gaikwad
110
Reputation points
commented 2024-07-03T08:41:38.42+00:00
Ashwini Gaikwad
110
Reputation points
3 answers
How to reduce unnecessary high memory usage in a Databricks cluster?
We are having unnecessary high memory usage even when nothing is running on the cluster. When the cluster first starts, it's fine, but when I run a script and it finishes executing, nothing gets back to the idle (initial) state (even hours after nothing…
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,045 questions
Azure Startups
Azure Startups
Azure: A cloud computing platform and infrastructure for building, deploying and managing applications and services through a worldwide network of Microsoft-managed datacenters.Startups: Companies that are in their initial stages of business and typically developing a business model and seeking financing.
236 questions
asked 2024-05-08T08:58:46.4433333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(144, 39%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESH%3C/text%3E%3C/svg%3E)
Senad Hadzikic
20
Reputation points
edited the question 2024-07-03T04:18:19.4633333+00:00
Ryan Hill
26,866
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Set cloudFiles.maxFileAge and cloudFiles.backfillInterval values in Autoloader
I'm using following in the autoloader options. .option("cloudFiles.maxFileAge", "90 days")\ .option("cloudFiles.backfillInterval", "1 day")\ Our data retention policy is 7 years. Shall I use maxFileAge as 7 years…
asked 2024-07-02T02:12:40.21+00:00
Hiran Amarathunga
65
Reputation points
accepted 2024-07-02T07:43:44.5933333+00:00
Hiran Amarathunga
65
Reputation points
3 answers
How to specify a custom catalog name for Azure Databricks Delta Lake Dataset in ADF
Hello, I am creating an Azure Databricks Delta Lake Dataset in ADF and I am only able to choose the database name that links to Databricks's hive_metastore. How can I specify a custom catalog name that I created in Databricks instead of…
asked 2024-01-04T06:25:53.11+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(249.60000000000002, 17%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ETY%3C/text%3E%3C/svg%3E)
Tom Young
0
Reputation points
commented 2024-07-02T04:23:57.05+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(105.60000000000001, 51%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EEP%3C/text%3E%3C/svg%3E)
Evgeniy Perfilyev
0
Reputation points
1 answer
One of the answers was accepted by the question author.
Serverless warehouse suddenly stops to start up.
Hey All. From today, suddenly we are getting below error while starting a serverless warehouse. Details for the latest failure: Error: Cluster launch timeout. Type: SERVICE_FAULT Code: K8S_DBR_CLUSTER_LAUNCH_TIMEOUT Warehouse details: Type:…
asked 2024-06-25T12:48:53.3833333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(278.4, 41%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBS%3C/text%3E%3C/svg%3E)
Bhargav Shir
20
Reputation points
commented 2024-07-02T04:16:33.1166667+00:00
PRADEEPCHEEKATLA-MSFT
84,371
Reputation points Microsoft Employee
1 answer
Can we run delta live tables with free tier azure account?
While running the delta live tables pipeline in azure databricks I'm getting an error saying: QuotaExceeded, error message: Operation could not be completed as it results in exceeding approved Total Regional Cores quota. Additional details - Deployment…
asked 2024-07-01T10:14:34.0733333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(48, 19%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKB%3C/text%3E%3C/svg%3E)
Kartik B
0
Reputation points
answered 2024-07-01T11:17:49.9766667+00:00
PRADEEPCHEEKATLA-MSFT
84,371
Reputation points Microsoft Employee
1 answer
How to solve Invalid SessionHandle error with Azure Databricks ?
I am applying a SQLDatabaseChain Chatbot model by using LangChain SQLDatabaseChain and GPT4. I first created this model on Azure Databricks notebook like this : import json import os import langchain import mlflow from mlflow.models import…
asked 2024-06-17T14:38:14.4333333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(19.2, 46%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EFY%3C/text%3E%3C/svg%3E)
Feng YIJUN
0
Reputation points
edited an answer 2024-07-01T07:42:48.6866667+00:00
Feng YIJUN
0
Reputation points
1 answer
How do I share all of my databricks notebooks with all databricks users?
Hi all, I know that I've done this in the distant past, but we have a new instance of Databricks and I need to do a global setting to share all of my notebooks with all Databricks users (read only). That way I don't need to remember to share individual…
asked 2024-06-27T13:57:35.7666667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(80, 15%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBK%3C/text%3E%3C/svg%3E)
Barb K
0
Reputation points
edited a comment 2024-07-01T05:28:58.0633333+00:00
PRADEEPCHEEKATLA-MSFT
84,371
Reputation points Microsoft Employee
1 answer
while creating cluster in databricks i am getting following error Azure Quota Exceeded Exception: Error code: SkuNotAvailable, error message: The requested VM size for resource 'Following SKUs have failed for Capacity Restrictions: Standard_DS3_v2' is cur
while creating cluster in databricks i am getting following error Azure Quota Exceeded Exception: Error code: SkuNotAvailable, error message: The requested VM size for resource 'Following SKUs have failed for Capacity Restrictions: Standard_DS3_v2' is…
asked 2024-06-28T12:09:26.57+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(249.60000000000002, 83%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
azurecloud
0
Reputation points
edited an answer 2024-07-01T04:37:52.52+00:00
PRADEEPCHEEKATLA-MSFT
84,371
Reputation points Microsoft Employee
1 answer
array in databricks
Hi , I got the 1st row in sorted order how can i get the view on row1,row2,row3 using array and how the reshuffling will happen 1,2,3,4,5 --- sorted order 5,1,2,3,4 - 1 4,5,1,2,3 - 2 3,4,5,1,2 - 3
asked 2024-06-25T08:20:36.5333333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 13%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVS%3C/text%3E%3C/svg%3E)
Vineet S
265
Reputation points
commented 2024-06-30T15:08:50.3966667+00:00
PRADEEPCHEEKATLA-MSFT
84,371
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Azure Databricks fails
Hello, In the databricks notebook which is provided by Microsoft training classes, when I tried to import => read a data (csv or json) like path = source + "/wikipedia/pagecounts/staging_parquet_en_only_clean/" files =…
asked 2023-07-20T13:58:55.4033333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(73.60000000000001, 91%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EC%3C/text%3E%3C/svg%3E)
Catherine 刘
30
Reputation points
edited a comment 2024-06-29T15:40:00.8866667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(22.400000000000002, 73%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECS%3C/text%3E%3C/svg%3E)
Choi, Seung-Rak
0
Reputation points
2 answers
One of the answers was accepted by the question author.
databricks cluster sizing
Hey, how to calculate cluster core and workers node of 10gb data load every 2 hours ... what is the calculation behind this
0 answers
How can I remove the `sample` catalog from Azure Databricks Workspaces?
All Azure Databricks Workspaces come with a sample catalog owned by Databricks that I cannot seem to remove or hide. I have tried dropping it, and I have also tried revoking and denying permissions on the catalog, but I keep receiving an error that is…
asked 2023-10-31T19:35:30.61+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(281.6, 28.999999999999996%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ETT%3C/text%3E%3C/svg%3E)
Tim Thein
5
Reputation points
commented 2024-06-28T04:06:26.55+00:00
PRADEEPCHEEKATLA-MSFT
84,371
Reputation points Microsoft Employee
1 answer
view in dataframe
hey, how we can create or replace view statement in spark sql in dataframe of databricks create or replace view as (select * from temp1)
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(12.8, 48%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EH%3C/text%3E%3C/svg%3E)