Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,218 questions
1 answer
One of the answers was accepted by the question author.
is there a spark connector for Azure Cosmos DB MongoDB API RU?
I'm using Azure Databricks to develop a proof of concept. Is there support for a Spark Connector for Azure Cosmos DB MongoDB API? If yes, what versions are supported. Is both batch and stream operations supported?
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(240, 22%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ELJ%3C/text%3E%3C/svg%3E)
1 answer
Databricks vs Batch pool
Hi, I have Googled the difference between using Azure Databrick and Batch pool to run python pipelines and ETL. But I haven’t found a clear difference between them two. Based on what I have Googled, Databrick can become very expensive, so cost matters a…
asked 2024-11-07T14:10:41.1766667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(112.00000000000001, 59%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ER%3C/text%3E%3C/svg%3E)
Renan
40
Reputation points
commented 2024-11-10T17:06:16.29+00:00
Prrudram-MSFT
25,321
Reputation points
1 answer
understanding how to use mergeSchema=True
In COPY INTO under FORMAT_OPTIONS generally i use mergeSchema=True but in few queries i am seeing that it is also metioned under copy_options why ? and I have also seen mergeSchema=true given under options while read or readStream, i have seen that in…
asked 2024-11-07T09:17:43.31+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(89.60000000000001, 42%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMS%3C/text%3E%3C/svg%3E)
Mantri Sai Ashish
0
Reputation points
edited an answer 2024-11-10T12:44:18.5766667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(201.6, 18%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGG%3C/text%3E%3C/svg%3E)
Ganesh Gurram
1,110
Reputation points Microsoft Vendor
1 answer
I dont see Account Console in Databricks. Please advise
I have personal account and when I log into databricks I dont see account console to admin stuff
asked 2024-10-31T13:46:23.6+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(259.20000000000005, 18%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGE%3C/text%3E%3C/svg%3E)
Gokhan Erkus
0
Reputation points
commented 2024-11-10T10:46:19.4733333+00:00
Vinodh247
23,346
Reputation points MVP
0 answers
How to read zip file in Azure Synapse Notebook with extension of .zip.
Hi Team, Could you please advise the pyspark code, how to read zip files in synapse notebook. Example: File_Name - samplefile.zip Note: In the samplefile.zip we have multiple csv files, please let us know. The file Extension is only with .zip. …
asked 2024-11-08T16:44:52.83+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(163.2, 22%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESM%3C/text%3E%3C/svg%3E)
SaiSekhar, MahasivaRavi (Philadelphia)
60
Reputation points
edited a comment 2024-11-08T18:31:27.7233333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 61%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EP%3C/text%3E%3C/svg%3E)
phemanth
11,280
Reputation points Microsoft Vendor
1 answer
One of the answers was accepted by the question author.
How to list secret scopes and identify key vaults
What is the best way to view all existing secret scopes within a Databricks workspace? Additionally, how can we identify the specific Azure Key Vault linked to each of these secret scopes? I’d like to understand the steps or API calls needed to retrieve…
asked 2024-11-06T11:50:02.7566667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(300.8, 15%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EG2%3C/text%3E%3C/svg%3E)
Gabriel-2005
325
Reputation points
accepted 2024-11-08T11:25:54.35+00:00
Gabriel-2005
325
Reputation points
1 answer
getting error while creating mount point in data bricks
Hi, I am getting the following error when trying to create mount point. py4j.security.Py4JSecurityException: Method public com.databricks.backend.daemon.dbutils.DBUtilsCore$Result…
asked 2024-10-24T09:28:58.9066667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(156.8, 77%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ELN%3C/text%3E%3C/svg%3E)
Lakshmi Narayana Sarma Bhamidipati
90
Reputation points
commented 2024-11-08T08:01:37.0633333+00:00
Lakshmi Narayana Sarma Bhamidipati
90
Reputation points
1 answer
Azure Databricks Workspace: Users are being added to the admins group on their own
We have an Azure Databricks workspace with Unity Catalog enabled. We are using SCIM provisioning to sync some Entra security groups to the workspace so that we can control access to the catalog items. We have noticed that some users are being added…
asked 2024-11-01T10:04:34.28+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(201.6, 72%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECO%3C/text%3E%3C/svg%3E)
Chris O'Brien
0
Reputation points
commented 2024-11-08T02:18:39.42+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(99.2, 65%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECB%3C/text%3E%3C/svg%3E)
Chandra Boorla
2,915
Reputation points Microsoft Vendor
2 answers
Want to know my subscription and billing details for azure account
Hi, I had created azure free trail subscription account on sep, 6th, 2024 and cancelled the subscription as well but today i got a mail regarding billing. when i proctored the billing information in the account there were no subscriptions listed. i…
asked 2024-11-06T06:01:05.6933333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(208, 43%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENK%3C/text%3E%3C/svg%3E)
Naveen Korra
0
Reputation points
commented 2024-11-08T01:15:31.19+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 47%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERP%3C/text%3E%3C/svg%3E)
Rahul Podila
195
Reputation points Microsoft Vendor
1 answer
One of the answers was accepted by the question author.
Availability of Serverless Compute in South African North region
Are there any plans and dates to bring Databricks of Serverless Compute to the South African North region? Thanks, Andreas.
asked 2024-11-07T02:51:10.8433333+00:00
Andreas Cambitsis
66
Reputation points
accepted 2024-11-07T06:43:48.1566667+00:00
Andreas Cambitsis
66
Reputation points
1 answer
One of the answers was accepted by the question author.
copy tables from sql server on-prem to .parquet format in delta lake
Currently our copy process for 10 tables is running in pipeline in ADF, our source is ON-PREM SQL Server and target is in BLOB Storage GEN2. it copies data(tables) in parquet file format in HOT storage Account --- StorageV2 (general purpose v2) but I…
asked 2024-10-29T19:03:53.5233333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(28.799999999999997, 10%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EZH%3C/text%3E%3C/svg%3E)
Zaeemul Haque
45
Reputation points
accepted 2024-11-06T16:58:39.44+00:00
Zaeemul Haque
45
Reputation points
1 answer
One of the answers was accepted by the question author.
Azure Databricks Compute Instance Quota Issue
Unable to create a compute instance in Azure Databricks due to a quota issue. The error message received is: This account may not have enough CPU cores to start a cluster. Contact your administrator to increase the limits. Additionally, it is noted…
asked 2024-10-30T17:40:56.9433333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(297.6, 63%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDK%3C/text%3E%3C/svg%3E)
Dmitriy Khodyrev
20
Reputation points
accepted 2024-11-06T13:30:19.3333333+00:00
Dmitriy Khodyrev
20
Reputation points
1 answer
Unable to access account console in daatabrics
Hello guys I created a storage account through my pay as u go subscription and lauched a premium tier workspace for databricks but i was trying to connect my workspace to metastore in unity catalog but i dont see my manage account button to access…
asked 2024-10-30T11:19:15.72+00:00
Mantri Sai Ashish
0
Reputation points
commented 2024-11-06T13:23:25.7366667+00:00
PRADEEPCHEEKATLA
90,226
Reputation points
2 answers
I don't have access to admin console of databricks even though I am admin user
I created azure account and launched databricks, I don't have access to admin console of databricks even though I am the only admin user
asked 2024-11-01T03:49:21.9833333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(16, 73%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ER%3C/text%3E%3C/svg%3E)
Ramya
0
Reputation points
commented 2024-11-06T00:54:38.9+00:00
Chandra Boorla
2,915
Reputation points Microsoft Vendor
1 answer
Error occurred when using ODBC driver with Workload Identity to connect to Azure Databricks Workspace from an AKS Pod deployment
Hello everybody. As per the title, I am trying to achieve the followings: Connect to an Azure Databricks Workspace, in order to execute an SQL Query, from an Azure Function running on an AKS Pod. Pod deployment is made using CI/CD pipelines with…
asked 2024-11-04T15:55:37.3633333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(163.2, 93%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDI%3C/text%3E%3C/svg%3E)
Dragos Ionita
0
Reputation points
commented 2024-11-05T16:03:46.5266667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(246.4, 4%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESS%3C/text%3E%3C/svg%3E)
Sina Salam
12,011
Reputation points
1 answer
One of the answers was accepted by the question author.
Azure Databricks is not sending email when Databricks Alert condition is met
Azure Databricks is not sending an email to the destination email address when Databricks Alert condition is met, Sometimes this is working and sometimes not.
asked 2024-11-01T09:50:13.5066667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(217.60000000000002, 84%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMV%3C/text%3E%3C/svg%3E)
Mekala, Vamshi
20
Reputation points
commented 2024-11-05T08:58:24.0766667+00:00
Mekala, Vamshi
20
Reputation points
0 answers
Autoloader in Synapse
I want to load data incrementally using Azure Synapse Analytics Notebook(dedicated spark pool) there is an option on databricks to load data incrementally which is autoloader so my question was can we use this option in Azure Synapse Analytics…
asked 2024-01-24T11:33:32.82+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(57.599999999999994, 69%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAS%3C/text%3E%3C/svg%3E)
Aditya Singh
160
Reputation points
edited the question 2024-11-05T03:52:58.6933333+00:00
Sumarigo-MSFT
47,021
Reputation points Microsoft Employee
2 answers
Error when creating Access Connector for Azure Databricks
So, just created storage, and would like to setup the access connector, and faces the below error. Anyone seen this before ? All insights are very much appreciated
asked 2024-10-31T10:38:45.89+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(19.2, 73%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAT%3C/text%3E%3C/svg%3E)
Anders Thøgersen
0
Reputation points
commented 2024-11-05T01:13:43.6766667+00:00
Chandra Boorla
2,915
Reputation points Microsoft Vendor
3 answers
Azure Databricks Compute error: This account may not have enough CPU cores to satisfy this request Estimated available: 0, requested: 4
I have created Azure Free Tier Subscription with a credit $200 a day ago and deployed Azure Databricks for 14 days free as well. In Quotas I have 4CPU free: But when I'm trying to create Compte in Azure Databricks it says This account may not have…
asked 2024-10-27T20:06:04.87+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(195.2, 30%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESV%3C/text%3E%3C/svg%3E)
Sergei Varaksin
0
Reputation points
answered 2024-11-04T06:15:09.8533333+00:00
Ganesh Gurram
1,110
Reputation points Microsoft Vendor
1 answer
One of the answers was accepted by the question author.
Automating conversion of python scripts to databricks notebook
I'm looking to convert a Python .py file into a format compatible with Databricks, where the file can be split into multiple cells without using the Databricks GUI. Eventually, I’d like to automate this process. What’s the minimal code needed to create a…
asked 2024-10-29T14:45:29.6666667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(99.2, 32%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES5%3C/text%3E%3C/svg%3E)
Stephen-5002
40
Reputation points
edited an answer 2024-11-04T04:08:48.58+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 8%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EST%3C/text%3E%3C/svg%3E)
Smaran Thoomu
16,890
Reputation points Microsoft Vendor