Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,226 questions
1 answer
One of the answers was accepted by the question author.
Executing mutiple queries in a single transaction of a notebook on databricks
I'm working on executing multiple queries within a single transaction on Databricks using Golang. However, I'm encountering a 'not implemented' error. Upon reviewing the library code, I found the following: // Not supported in Databricks. func (c…
asked 2024-11-15T07:43:37.5033333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(300.8, 15%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EG2%3C/text%3E%3C/svg%3E)
Gabriel-2005
325
Reputation points
accepted 2024-11-16T07:35:36.8066667+00:00
Gabriel-2005
325
Reputation points
0 answers
Table Access control option is not available in Azure Databricks Compute cluster
Hi, I am trying to enable Table Access control in one of my Azure Databricks Compute Cluster, the cluster was created around 02 years back with pricing tier selected as Standard, now we have upgraded the same cluster to Premium tier in order to…
asked 2024-11-15T15:16:40.88+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(124.80000000000001, 24%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESK%3C/text%3E%3C/svg%3E)
Satish Kumar
51
Reputation points
commented 2024-11-15T19:28:49.5+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(201.6, 18%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGG%3C/text%3E%3C/svg%3E)
Ganesh Gurram
1,350
Reputation points Microsoft Vendor
1 answer
How to downgrade from TLS 1.3 to 1.2 in Azure Databricks env
Hi, As part of troubleshooting a current intermittent issue with connecting to an external API, I am being asked to downgrade the TLS version from 1.3 to 1.2. I am running a python notebook in our Azure Databricks environment and believe that TLS 1.3 is…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(6.4, 50%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3E_%3C/text%3E%3C/svg%3E)
0 answers
Create views in Azure Databricks Unity catalog using service principal
Hello Team, Is there a possibility to Create views in Azure databricks Unity Catalog using service principal. and if there is how to update views based on tables added or deleted in schema. For e.g. I would like to create each views per tables in the…
asked 2024-11-13T16:51:42.7266667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(51.2, 77%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAG%3C/text%3E%3C/svg%3E)
Ashwini Gaikwad
130
Reputation points
commented 2024-11-15T12:49:37.1866667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 61%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EP%3C/text%3E%3C/svg%3E)
phemanth
11,450
Reputation points Microsoft Vendor
1 answer
Retrieving 'Person or Group' Fields from SharePoint List in Azure Data Factory or Synapse
I'm able to retrieve data from a SharePoint list using Azure Data Factory > Copy Data (via the SharePoint Connector), but the list contains 'Person or Group' type fields that return only 'Id' values in the results. These 'Id' values are not useful,…
asked 2024-11-12T14:14:03.32+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(115.19999999999999, 82%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESP%3C/text%3E%3C/svg%3E)
Sunny Punalkar
0
Reputation points
commented 2024-11-15T08:18:10.0766667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(252.8, 5%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ER%3C/text%3E%3C/svg%3E)
RaytheonXie_MSFT
36,256
Reputation points Microsoft Vendor
1 answer
How can i migrate jobs from one workspace to other workspace (Dev to QA)
How can i migrate jobs from one workspace to other workspace (Dev to QA) .
asked 2024-11-15T05:44:35.18+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(156.8, 77%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ELN%3C/text%3E%3C/svg%3E)
Lakshmi Narayana Sarma Bhamidipati
90
Reputation points
commented 2024-11-15T06:25:50.3566667+00:00
Lakshmi Narayana Sarma Bhamidipati
90
Reputation points
0 answers
Connect to metastore table in databricks from other databricks notebook using service principal
I want to read table present in other databricks account in my databricks account using service principal. I am not able to find service principal related guide to connect to other databricks workspace. I can only see storage account related service…
asked 2024-11-14T10:50:15.56+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(99.2, 15%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBR%3C/text%3E%3C/svg%3E)
Bhargav R
0
Reputation points
commented 2024-11-15T06:17:30.5466667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(99.2, 65%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECB%3C/text%3E%3C/svg%3E)
Chandra Boorla
3,230
Reputation points Microsoft Vendor
1 answer
'manage account' > 'account console' option
db-q-1.JPGHello Team, I do have an access as Global Administrator, but I am not getting 'manage account' > 'account console' option in Azure Databricks @ top right corner. I need to create Unity Catalog metastore. Thanks
asked 2024-11-11T10:54:30.5666667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(179.20000000000002, 23%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAV%3C/text%3E%3C/svg%3E)
Ankur Varshney
0
Reputation points
commented 2024-11-14T18:01:14.67+00:00
phemanth
11,450
Reputation points Microsoft Vendor
1 answer
Databricks vs Batch pool
Hi, I have Googled the difference between using Azure Databrick and Batch pool to run python pipelines and ETL. But I haven’t found a clear difference between them two. Based on what I have Googled, Databrick can become very expensive, so cost matters a…
asked 2024-11-07T14:10:41.1766667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(112.00000000000001, 59%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ER%3C/text%3E%3C/svg%3E)
Renan
40
Reputation points
commented 2024-11-14T12:18:50.3+00:00
Prrudram-MSFT
25,641
Reputation points
1 answer
Azure Databricks Workspace: Users are being added to the admins group on their own
We have an Azure Databricks workspace with Unity Catalog enabled. We are using SCIM provisioning to sync some Entra security groups to the workspace so that we can control access to the catalog items. We have noticed that some users are being added…
asked 2024-11-01T10:04:34.28+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(201.6, 72%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECO%3C/text%3E%3C/svg%3E)
Chris O'Brien
0
Reputation points
commented 2024-11-12T01:07:33.7933333+00:00
Chandra Boorla
3,230
Reputation points Microsoft Vendor
1 answer
Configured privacy settings disallow access for workspace <workspace-id> over your current network.
Hi All, We have a couple of Databricks workspaces that are deployed with secure access - basically you can only log in when you are within the network, However, I am running into a problem where I sometimes get this error: “Configured privacy settings…
asked 2023-06-11T20:26:26.69+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(28.799999999999997, 57.00000000000001%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJA%3C/text%3E%3C/svg%3E)
John Aherne
141
Reputation points
commented 2024-11-11T15:08:53.56+00:00
Felippe Borges
81
Reputation points
1 answer
understanding how to use mergeSchema=True
In COPY INTO under FORMAT_OPTIONS generally i use mergeSchema=True but in few queries i am seeing that it is also metioned under copy_options why ? and I have also seen mergeSchema=true given under options while read or readStream, i have seen that in…
asked 2024-11-07T09:17:43.31+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(89.60000000000001, 42%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMS%3C/text%3E%3C/svg%3E)
Mantri Sai Ashish
0
Reputation points
commented 2024-11-11T12:02:13.4066667+00:00
Ganesh Gurram
1,350
Reputation points Microsoft Vendor
2 answers
getting error while creating mount point in data bricks
Hi, I am getting the following error when trying to create mount point. py4j.security.Py4JSecurityException: Method public com.databricks.backend.daemon.dbutils.DBUtilsCore$Result…
asked 2024-10-24T09:28:58.9066667+00:00
Lakshmi Narayana Sarma Bhamidipati
90
Reputation points
commented 2024-11-11T09:18:11.33+00:00
Chandra Boorla
3,230
Reputation points Microsoft Vendor
1 answer
One of the answers was accepted by the question author.
is there a spark connector for Azure Cosmos DB MongoDB API RU?
I'm using Azure Databricks to develop a proof of concept. Is there support for a Spark Connector for Azure Cosmos DB MongoDB API? If yes, what versions are supported. Is both batch and stream operations supported?
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(240, 22%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ELJ%3C/text%3E%3C/svg%3E)
1 answer
I dont see Account Console in Databricks. Please advise
I have personal account and when I log into databricks I dont see account console to admin stuff
asked 2024-10-31T13:46:23.6+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(259.20000000000005, 18%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGE%3C/text%3E%3C/svg%3E)
Gokhan Erkus
0
Reputation points
commented 2024-11-10T10:46:19.4733333+00:00
Vinodh247
23,756
Reputation points MVP
0 answers
How to read zip file in Azure Synapse Notebook with extension of .zip.
Hi Team, Could you please advise the pyspark code, how to read zip files in synapse notebook. Example: File_Name - samplefile.zip Note: In the samplefile.zip we have multiple csv files, please let us know. The file Extension is only with .zip. …
asked 2024-11-08T16:44:52.83+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(163.2, 22%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESM%3C/text%3E%3C/svg%3E)
SaiSekhar, MahasivaRavi (Philadelphia)
60
Reputation points
commented 2024-11-08T18:28:51.7833333+00:00
phemanth
11,450
Reputation points Microsoft Vendor
1 answer
One of the answers was accepted by the question author.
How to list secret scopes and identify key vaults
What is the best way to view all existing secret scopes within a Databricks workspace? Additionally, how can we identify the specific Azure Key Vault linked to each of these secret scopes? I’d like to understand the steps or API calls needed to retrieve…
asked 2024-11-06T11:50:02.7566667+00:00
Gabriel-2005
325
Reputation points
accepted 2024-11-08T11:25:54.35+00:00
Gabriel-2005
325
Reputation points
2 answers
Want to know my subscription and billing details for azure account
Hi, I had created azure free trail subscription account on sep, 6th, 2024 and cancelled the subscription as well but today i got a mail regarding billing. when i proctored the billing information in the account there were no subscriptions listed. i…
asked 2024-11-06T06:01:05.6933333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(208, 43%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENK%3C/text%3E%3C/svg%3E)
Naveen Korra
0
Reputation points
commented 2024-11-08T01:15:31.19+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 47%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERP%3C/text%3E%3C/svg%3E)
Rahul Podila
350
Reputation points Microsoft Vendor
1 answer
One of the answers was accepted by the question author.
Availability of Serverless Compute in South African North region
Are there any plans and dates to bring Databricks of Serverless Compute to the South African North region? Thanks, Andreas.
asked 2024-11-07T02:51:10.8433333+00:00
Andreas Cambitsis
66
Reputation points
accepted 2024-11-07T06:43:48.1566667+00:00
Andreas Cambitsis
66
Reputation points
1 answer
One of the answers was accepted by the question author.
copy tables from sql server on-prem to .parquet format in delta lake
Currently our copy process for 10 tables is running in pipeline in ADF, our source is ON-PREM SQL Server and target is in BLOB Storage GEN2. it copies data(tables) in parquet file format in HOT storage Account --- StorageV2 (general purpose v2) but I…
asked 2024-10-29T19:03:53.5233333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(28.799999999999997, 10%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EZH%3C/text%3E%3C/svg%3E)
Zaeemul Haque
45
Reputation points
accepted 2024-11-06T16:58:39.44+00:00
Zaeemul Haque
45
Reputation points