Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,011 questions
1 answer
Admin portal option is not visible in my account
Admin portal option is not visible in my account. Could you please assist me.
asked 2024-06-12T19:08:22.1166667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(76.8, 3%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ED%3C/text%3E%3C/svg%3E)
Deepak2024
0
Reputation points
answered 2024-06-12T19:45:21.4466667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(291.2, 38%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMP%3C/text%3E%3C/svg%3E)
Marcin Policht
15,370
Reputation points MVP
0 answers
Restricting files/folders to upload into External volumes in Azure databricks UC workspace
Hello Team, Is there a way to restrict the files or folders to upload/download from external volumes same like DBFS? Is there any option to disable the uploading files/folders feature in external volumes of azure databricks workspace with Unity Catalog.…
asked 2024-06-06T17:37:57.5833333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 24%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAG%3C/text%3E%3C/svg%3E)
Ashwini Gaikwad
85
Reputation points
commented 2024-06-12T16:09:13.3533333+00:00
Ashwini Gaikwad
85
Reputation points
0 answers
Run Databricks notebook from ADF - error to find azure module to save the data in blob storage
Hi Guys, The requirement is - Call Rest API, read the records in jsonlines format and load into table in Azure SQL server. I used Databricks to read the jsonlines from Open API using Python script. It can read and keep the data into a file in Azure blob…
asked 2024-06-12T12:39:10.24+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(284.8, 35%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESS%3C/text%3E%3C/svg%3E)
Sarmistha Sarkar
0
Reputation points
asked 2024-06-12T12:39:10.24+00:00
Sarmistha Sarkar
0
Reputation points
1 answer
Issues while writing into bad_records path in Databricks
Hello All, I would like to get your inputs with a scenario that I see while writing into the bad_records file. I am reading a ‘Ԓ’ delimited CSV file based on a schema that I have already defined. I have enabled error handling while reading the file to…
asked 2024-06-05T01:13:41.47+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(38.4, 78%, 13%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAT%3C/text%3E%3C/svg%3E)
Alok Thampi
91
Reputation points
answered 2024-06-12T12:06:38.5633333+00:00
Alok Thampi
91
Reputation points
0 answers
How do I assign managed identity to a databricks resource?
I have created a linked service from ADF to Databricks cluster, following https://techcommunity.microsoft.com/t5/azure-data-factory-blog/azure-databricks-activities-now-support-managed-identity/ba-p/1922818 I have provided contributor access to the ADF…
Azure
Azure
A cloud computing platform and infrastructure for building, deploying and managing applications and services through a worldwide network of Microsoft-managed datacenters.
1,037 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,011 questions
asked 2024-06-12T10:24:08.21+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(313.6, 81%, 40%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAK%3C/text%3E%3C/svg%3E)
Alexa Kiss
0
Reputation points Microsoft Employee
edited the question 2024-06-12T10:24:36.48+00:00
Alexa Kiss
0
Reputation points Microsoft Employee
2 answers
One of the answers was accepted by the question author.
When creating a second external location to the same path in Azure Databricks Unity Catalog it gives conflicting error for path. Is there any way to solve this?
Hello Team, When creating a second external location/external volumes to the same path with different folder or to the root location gives an error see below for details in Azure Databricks Unity Catalog as it gives conflict error for path. Is there any…
asked 2024-06-04T10:23:22.8733333+00:00
Ashwini Gaikwad
85
Reputation points
commented 2024-06-12T09:37:40.13+00:00
PRADEEPCHEEKATLA-MSFT
81,646
Reputation points Microsoft Employee
1 answer
The scim API is by default adding users to admins group in azure databricks
Hi, When we are invoking scim API in azure databricks it is by default adding users to the admins group and also after deleting users from only admins group they are being created again. Also calling scim API with adding groups as users also adding them…
asked 2024-06-05T11:29:19.43+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(262.40000000000003, 26%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGN%3C/text%3E%3C/svg%3E)
Gupta, Neha
0
Reputation points
commented 2024-06-12T07:15:58.77+00:00
PRADEEPCHEEKATLA-MSFT
81,646
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Connect to Blob storage from Azure Databricks SQL
So I would like to read a table from a CSV file on Azure Blob Storage in my own account, and load it into a table in Unity Catalog on databricks (hopefully using SQL). I have tried this SQL command: CREATE TABLE IF NOT EXISTS <table_name>; COPY…
asked 2024-06-11T13:36:31.3633333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(217.60000000000002, 47%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKW%3C/text%3E%3C/svg%3E)
Kaizad Wadia
0
Reputation points
commented 2024-06-12T06:52:52.87+00:00
PRADEEPCHEEKATLA-MSFT
81,646
Reputation points Microsoft Employee
1 answer
org.apache.hadoop.fs.FileAlreadyExistsException: Failed to rename temp file
[Repeat Question due to old thread] We have built a streaming pipeline with spark autoloader. Source Folder is a azure blob container. We've encountered a rare issue (could not replicate it). Below is the exception…
asked 2024-06-07T01:40:56.47+00:00
Hiran Amarathunga
5
Reputation points
commented 2024-06-12T05:15:04.35+00:00
PRADEEPCHEEKATLA-MSFT
81,646
Reputation points Microsoft Employee
0 answers
Azure Databricks exercise error
Keep receiving the error "No such file or directory /your_correct_source_value/wikipedia/pagecounts/staging_parquet_en_only_clean" When I checked Wikipedia, it appears this dataset has been deprecated since 2016-08-01 Could a new dataset be…
asked 2024-06-05T19:59:42.4166667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(291.2, 46%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJO%3C/text%3E%3C/svg%3E)
Joab Odera
0
Reputation points
commented 2024-06-12T05:07:00.8033333+00:00
PRADEEPCHEEKATLA-MSFT
81,646
Reputation points Microsoft Employee
1 answer
What Azure service should I use to deploy a complex Python program on the cloud?
Background<br> I have developed a Python program that fetches data from three different REST APIs, processes it, and inserts it into a database. The program also queries the database to identify which values to fetch from the APIs, so there is…
asked 2024-06-05T13:06:56.45+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(310.4, 2%, 40%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EK1%3C/text%3E%3C/svg%3E)
kman-1604
0
Reputation points
commented 2024-06-12T05:05:24.94+00:00
PRADEEPCHEEKATLA-MSFT
81,646
Reputation points Microsoft Employee
1 answer
deploying Azure databrick with datalake
Deploying Azure Databricks creates an additional resource group in the background, which includes a data lake. Is it possible to use the data lake that I have already deployed in Azure instead of the one provisioned by Azure Databricks?
asked 2024-06-05T08:40:13.1866667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(80, 52%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
Sujeet
0
Reputation points
commented 2024-06-12T05:04:01.3966667+00:00
PRADEEPCHEEKATLA-MSFT
81,646
Reputation points Microsoft Employee
1 answer
Integrating Databricks notebooks in Azure ML using SDK V2
Hi all, We currently have some Azure Databricks notebooks in production which we would like to integrate in Azure ML using the v2 SDK. I found resources to integrate these notebooks using the databricks_step in the v1 SDK. The official documentation…
asked 2024-06-07T13:12:55.3333333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(259.20000000000005, 18%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
Alexander
0
Reputation points
commented 2024-06-11T09:30:30.9966667+00:00
Amira Bedhiafi
17,706
Reputation points
1 answer
One of the answers was accepted by the question author.
Databricks Dev/Prod setup
We are a data team of 4 people. To make the process easy and more productive. Can we separate dev/prod environments at Databricks catalogue level rather than the workspace level? Can anyone share any thoughts on this? Thanks
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(150.4, 14.000000000000002%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EV%3C/text%3E%3C/svg%3E)
1 answer
One of the answers was accepted by the question author.
My Dev, test, prod environments are in different resource groups of same subscription. How do I create a devops pipeline in this case?a DevOps pipeline to deploy a
Hi, My dev, test and prod environments are in different resource groups of the same subscription. I am involved in a data engineering project where I will be using primarily below resources - ADLS - data storage ADF - Orchestration Azure Databricks - QC…
asked 2024-05-14T19:23:38.4366667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(70.4, 90%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EST%3C/text%3E%3C/svg%3E)
Shashwat Tiwary
40
Reputation points
accepted 2024-06-10T09:53:29.01+00:00
Shashwat Tiwary
40
Reputation points
1 answer
Azure Data Bricks - User Doesn't have permission to perform this action while connecting to Azure Synapse Dedicate Pool
We are connecting Azure Synapse Analytics - Dedicated Pool using the PySpark Code that runs from Azure Data Bricks using SQL Authentication. While running, we are getting the below error when we use a user with db_datawriter and db_datareader…
asked 2024-06-07T11:26:00.4933333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(230.39999999999998, 97%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPS%3C/text%3E%3C/svg%3E)
Praveen Sreeram
1
Reputation point
commented 2024-06-10T09:16:26.0533333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 8%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EST%3C/text%3E%3C/svg%3E)
Smaran Thoomu
11,370
Reputation points Microsoft Vendor
2 answers
How to ignore the records in ADF Data Flows
Hi All I am building a data transamination using mapping data flows ,I have a time stamp field Like TimeStampUpdated in the target table. I want to lockup historical data with incremental data transamination and ignore the records coming in the…
asked 2024-05-23T06:58:12.53+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 6%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVR%3C/text%3E%3C/svg%3E)
venkat rao
65
Reputation points
commented 2024-06-10T08:32:00.3+00:00
PRADEEPCHEEKATLA-MSFT
81,646
Reputation points Microsoft Employee
1 answer
How can i connect Azure Databricks to Neo4j??
Hello, I want to connect to neo4j from Azure Databricks. What are the different approaches do I have? I am trying to connect here and i getting following error. Do I need to do anything before running the code? i mean setup managed identity or enable…
asked 2024-06-10T03:41:31.9866667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(201.6, 73%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESR%3C/text%3E%3C/svg%3E)
Siddartha Reddy Jammula
20
Reputation points
answered 2024-06-10T07:23:32.1533333+00:00
PRADEEPCHEEKATLA-MSFT
81,646
Reputation points Microsoft Employee
1 answer
"Premium Automated Serverless Compute - Promo DBU" expenses arise from what, how can I disable it, and why are the costs so high?
"Premium Automated Serverless Compute - Promo DBU" expenses arise from what, how can I disable it, and why are the costs so high? detail in below
asked 2024-06-09T16:43:36.0033333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(70.4, 0%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPT%3C/text%3E%3C/svg%3E)
Pratya Thanwatthanakit
0
Reputation points
edited an answer 2024-06-10T06:43:03.4633333+00:00
PRADEEPCHEEKATLA-MSFT
81,646
Reputation points Microsoft Employee
0 answers
Databricks Simba Spark ODBC .NET8 C# Driver Parameters in SQL Queries
Hello, I'm using Simba ODBC driver v2.8.0 in order to query data from my azure databrick sql warehouse into a .net 8 Asp.net Api App. The ODBC driver works fine using plain text query but i need to parametrize the query. Searching around I found that it…
asked 2024-06-03T15:30:12.4666667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(265.6, 63%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ELN%3C/text%3E%3C/svg%3E)
Luigi Navarra
5
Reputation points
commented 2024-06-10T06:05:14.48+00:00
PRADEEPCHEEKATLA-MSFT
81,646
Reputation points Microsoft Employee