Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,070 questions
1 answer
Append in Liquid Cluster enabled table is not completing on DBR 15.3 version
I am trying do analysis with a Partition Table and Liquid Clustered table. As per Azure Databricks recommendation, I am using DBR 15.2 to execute the code. I have created a clustered table as and using an append operation which is specified below. Few…
asked 2024-07-13T07:29:40.88+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 35%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESG%3C/text%3E%3C/svg%3E)
Sudipta Goswami
20
Reputation points
commented 2024-07-16T17:09:56.4566667+00:00
Sudipta Goswami
20
Reputation points
0 answers
CORS Issues Between Azure Static Web Apps and Azure Databricks
Hi, I'm currently facing some CORS issues in my application setup and would like to know your opinion on how to solve them. Front-end: Angular application deployed on Azure Static Web Apps (.azurestaticapps.net) Endpoint to access: Model serving…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(44.800000000000004, 69%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERN%3C/text%3E%3C/svg%3E)
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(224.00000000000003, 86%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
1 answer
How to replace Data flow activity having transformations inside it in ADF with other activity.
I have Azure data factory pipeline which have data flow activity. Data flow activity points to source file in storage account gets data from it as a source then performs different transformations on data using conditional split, derived column, flatten…
asked 2024-07-11T20:07:42.1866667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(67.2, 12%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERJ%3C/text%3E%3C/svg%3E)
Rahi Jangle
0
Reputation points
commented 2024-07-16T05:32:45.37+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(12.8, 48%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EH%3C/text%3E%3C/svg%3E)
Harishga
5,910
Reputation points Microsoft Vendor
2 answers
Managed storage account's compliance
Azure Databricks managed storage accounts need to have the key access disabled. But since these have deny assignment, I am able to see / influence the configuration. How do I make these storage accounts be green for this compliance?
asked 2024-06-19T17:48:49.89+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(160, 46%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECG%3C/text%3E%3C/svg%3E)
Chitra Gurumurthy (NON EA SC ALT)
0
Reputation points Microsoft Employee
commented 2024-07-16T04:52:20.72+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(150.4, 64%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENR%3C/text%3E%3C/svg%3E)
Nehruji R
4,451
Reputation points Microsoft Vendor
2 answers
What is the best way to access data in the data bricks by using azure function?
I just tried to load data from data bricks by using data bricks jobs API and azure function. Can I know is there another way to do the same thing that based on azure function?
asked 2024-07-12T09:08:56.7366667+00:00
Athula Chandrawansha
20
Reputation points
commented 2024-07-15T14:12:14.23+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(86.4, 36%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBM%3C/text%3E%3C/svg%3E)
Bhargava-MSFT
28,951
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
In Databricks, how to remove duplicates from a column of type array<array<string>>
I'm doing a select collect_set on a field in an array of struct type which results in a column of type array<array<strings>>. I would like to get the distinct strings in each of the nested arrays. How do I do this? The function…
asked 2024-07-13T06:28:22.3266667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(176, 31%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESW%3C/text%3E%3C/svg%3E)
Sara Watson
20
Reputation points
accepted 2024-07-13T06:30:40.57+00:00
Sara Watson
20
Reputation points
1 answer
One of the answers was accepted by the question author.
how to manage the data plane of Azure databricks
Normally, how to manage Azure databricks workspace and the resources, objects and services controlled by the workspace
asked 2024-02-27T19:05:49.3766667+00:00
Chen, Albert G
46
Reputation points
commented 2024-07-12T20:09:26.7233333+00:00
Chen, Albert G
46
Reputation points
1 answer
Cannot create metastore in Databricks.
Hi Community, I am having an issue when creating a metastore in Databricks account. I used to have one working, but I deleted it. Now I need to create one and the system keeps telling me that the current region already contains a metastore. That is not…
asked 2023-11-05T05:59:13.7+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(198.4, 39%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVV%3C/text%3E%3C/svg%3E)
Victoriano Vega
0
Reputation points
commented 2024-07-12T18:20:46.8033333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(64, 83%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERL%3C/text%3E%3C/svg%3E)
Rakib Laskar
0
Reputation points
2 answers
One of the answers was accepted by the question author.
Want to migrate from one Synapse workspace to another Synapse workspace
Hi team, Want to migrate from one Synapse workspace to another Synapse workspace. One being Dev environment, another one test environment. Please provide leads. Regards, NagaSri
asked 2021-10-18T15:17:49.867+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(198.4, 80%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EN%3C/text%3E%3C/svg%3E)
Naga
66
Reputation points
answered 2024-07-12T15:07:38.5833333+00:00
Mehdi Belkhiria
0
Reputation points Microsoft Employee
0 answers
primary key in adf pipeline
ADF package is failing due to Primarky and not null column available in target table due to which pipiline is failing how to handle such situation in adf.. Please prrimary key is only creating unique constraint
1 answer
whitelist the serverless data plane subnets in the cloud region
I am following below instructions from the documentation to whitelist the serverless data plane subnets in the cloud region of your Databricks workspace. But unable to find ARM resource ID of the serverless compute subnet details …
asked 2024-07-10T05:35:31.7966667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(25.6, 17%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESS%3C/text%3E%3C/svg%3E)
Santhosh Singh (ext)
0
Reputation points
commented 2024-07-12T05:30:06.1266667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 8%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EST%3C/text%3E%3C/svg%3E)
Smaran Thoomu
12,610
Reputation points Microsoft Vendor
1 answer
Azure Databricks - Timeout error after 60 minutes when launching an Azure Databricks cluster
When I attempt to start a cluster through the Azure Databricks portal/UI, after 30 minutes I receive the following error in the event log: Failed to add 3 containers to the compute. Will attempt retry: true. Reason: Cloud provider launch failure Azure…
asked 2024-07-12T05:04:06.92+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(284.8, 11%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMP%3C/text%3E%3C/svg%3E)
Michael Pugliese
0
Reputation points
answered 2024-07-12T05:16:28.47+00:00
PRADEEPCHEEKATLA-MSFT
85,346
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
create the dateid table in sql server
Hi , how to populate the dateid and date in dimtime table using script as below output upto 2029 excluding saturday and sunday create table table (dateid int,day varchar(23), dateid | day 20240601 | 2024 June 01 20290601 | 2029 June…
2 answers
Azure Databricks Billing
I am confused about how the databricks service is billed under Azure. From documentation, it is said that Databricks is totally integrated with Azure billing: one bill for both Azure infrastructure (VM, storage, Network traffic, etc) and Databricks…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(262.40000000000003, 27%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPJ%3C/text%3E%3C/svg%3E)
1 answer
merge statement in 2 data frame
Hi , how to use merge statement in 2 dataframe df1=spark.sql("" sellect cole1,col2 from table1""") df2=spark.sql("" sellect cole1,col2 from table2""") expected results merge into table2 using tabl1 on…
asked 2024-07-04T19:49:06.46+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 13%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVS%3C/text%3E%3C/svg%3E)
Vineet S
425
Reputation points
commented 2024-07-09T10:53:09.3966667+00:00
Smaran Thoomu
12,610
Reputation points Microsoft Vendor
1 answer
merge in sql server table using dataframe
how to use merge statement in 2 dataframe df1=spark.sql("" sellect cole1,col2 from table1""") Table2 from sql server
asked 2024-07-06T05:53:45.0133333+00:00
Vineet S
425
Reputation points
commented 2024-07-09T10:49:22.32+00:00
Smaran Thoomu
12,610
Reputation points Microsoft Vendor
1 answer
DataBricks Unity Catalog Lineage
Hi, I'm looking for support on the Databricks Unity Catalog (on the data lineage). So I'm trying to establish lineage between 2 schemas (with 50 odd tables within each schema). Data for the first schema is fetched from source files (via ADF pipeline),…
asked 2024-06-20T06:15:49.5666667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(307.2, 34%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESS%3C/text%3E%3C/svg%3E)
S, Santhosh M
0
Reputation points
commented 2024-07-09T10:34:25.3466667+00:00
PRADEEPCHEEKATLA-MSFT
85,346
Reputation points Microsoft Employee
1 answer
Guidance on how to use Service Principal with Certificate to Authorize for EventHub Stream Read
I found this documentation https://github.com/Azure/azure-event-hubs-spark/blob/master/docs/use-aad-authentication-to-connect-eventhubs.md online on how to use service principal with certificate to use spark stream read from EventHubs, I want to do this…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(67.2, 28.999999999999996%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EB%3C/text%3E%3C/svg%3E)
4 answers
How to reduce unnecessary high memory usage in a Databricks cluster?
We are having unnecessary high memory usage even when nothing is running on the cluster. When the cluster first starts, it's fine, but when I run a script and it finishes executing, nothing gets back to the idle (initial) state (even hours after nothing…
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,070 questions
Azure Startups
Azure Startups
Azure: A cloud computing platform and infrastructure for building, deploying and managing applications and services through a worldwide network of Microsoft-managed datacenters.Startups: Companies that are in their initial stages of business and typically developing a business model and seeking financing.
239 questions
asked 2024-05-08T08:58:46.4433333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(144, 39%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESH%3C/text%3E%3C/svg%3E)
Senad Hadzikic
20
Reputation points
commented 2024-07-08T17:44:30.65+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(278.4, 62%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
Alex
0
Reputation points
1 answer
One of the answers was accepted by the question author.
While running SQL query in Azure Databricks workspace i.e. on SQL warehouse as well as on UC enabled shared cluster facing an SSL handshake error
Hello Team, We have UC enabled Azure databricks workspace, also the Public access and delta sharing is disabled on our workspace. So while running the below SQL query on SQL Warehouse as well as on UC enabled shared cluster, I am receiving an…
asked 2024-06-28T10:06:18.7566667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 24%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAG%3C/text%3E%3C/svg%3E)
Ashwini Gaikwad
130
Reputation points
accepted 2024-07-08T11:09:53.34+00:00
Ashwini Gaikwad
130
Reputation points