Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,038 questions
1 answer
How can i connect Azure Databricks to Neo4j??
Hello, I want to connect to neo4j from Azure Databricks. What are the different approaches do I have? I am trying to connect here and i getting following error. Do I need to do anything before running the code? i mean setup managed identity or enable…
asked 2024-06-10T03:41:31.9866667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(201.6, 73%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESR%3C/text%3E%3C/svg%3E)
Siddartha Reddy Jammula
20
Reputation points
commented 2024-06-14T06:43:53.42+00:00
PRADEEPCHEEKATLA-MSFT
83,886
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
org.apache.hadoop.fs.FileAlreadyExistsException: Failed to rename temp file
[Repeat Question due to old thread] We have built a streaming pipeline with spark autoloader. Source Folder is a azure blob container. We've encountered a rare issue (could not replicate it). Below is the exception…
asked 2024-06-07T01:40:56.47+00:00
Hiran Amarathunga
45
Reputation points
accepted 2024-06-14T02:26:30.36+00:00
Hiran Amarathunga
45
Reputation points
1 answer
Run Databricks notebook from ADF - error to find azure module to save the data in blob storage
Hi Guys, The requirement is - Call Rest API, read the records in jsonlines format and load into table in Azure SQL server. I used Databricks to read the jsonlines from Open API using Python script. It can read and keep the data into a file in Azure blob…
asked 2024-06-12T12:39:10.24+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(284.8, 35%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESS%3C/text%3E%3C/svg%3E)
Sarmistha Sarkar
0
Reputation points
answered 2024-06-13T05:37:40.67+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 61%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EP%3C/text%3E%3C/svg%3E)
phemanth
7,825
Reputation points Microsoft Vendor
2 answers
One of the answers was accepted by the question author.
When creating a second external location to the same path in Azure Databricks Unity Catalog it gives conflicting error for path. Is there any way to solve this?
Hello Team, When creating a second external location/external volumes to the same path with different folder or to the root location gives an error see below for details in Azure Databricks Unity Catalog as it gives conflict error for path. Is there any…
asked 2024-06-04T10:23:22.8733333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 24%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAG%3C/text%3E%3C/svg%3E)
Ashwini Gaikwad
110
Reputation points
commented 2024-06-12T09:37:40.13+00:00
PRADEEPCHEEKATLA-MSFT
83,886
Reputation points Microsoft Employee
1 answer
The scim API is by default adding users to admins group in azure databricks
Hi, When we are invoking scim API in azure databricks it is by default adding users to the admins group and also after deleting users from only admins group they are being created again. Also calling scim API with adding groups as users also adding them…
asked 2024-06-05T11:29:19.43+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(262.40000000000003, 26%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGN%3C/text%3E%3C/svg%3E)
Gupta, Neha
0
Reputation points
commented 2024-06-12T07:15:58.77+00:00
PRADEEPCHEEKATLA-MSFT
83,886
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Connect to Blob storage from Azure Databricks SQL
So I would like to read a table from a CSV file on Azure Blob Storage in my own account, and load it into a table in Unity Catalog on databricks (hopefully using SQL). I have tried this SQL command: CREATE TABLE IF NOT EXISTS <table_name>; COPY…
asked 2024-06-11T13:36:31.3633333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(217.60000000000002, 47%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKW%3C/text%3E%3C/svg%3E)
Kaizad Wadia
20
Reputation points
commented 2024-06-12T06:52:52.87+00:00
PRADEEPCHEEKATLA-MSFT
83,886
Reputation points Microsoft Employee
1 answer
What Azure service should I use to deploy a complex Python program on the cloud?
Background<br> I have developed a Python program that fetches data from three different REST APIs, processes it, and inserts it into a database. The program also queries the database to identify which values to fetch from the APIs, so there is…
asked 2024-06-05T13:06:56.45+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(310.4, 2%, 40%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EK1%3C/text%3E%3C/svg%3E)
kman-1604
0
Reputation points
commented 2024-06-12T05:05:24.94+00:00
PRADEEPCHEEKATLA-MSFT
83,886
Reputation points Microsoft Employee
1 answer
deploying Azure databrick with datalake
Deploying Azure Databricks creates an additional resource group in the background, which includes a data lake. Is it possible to use the data lake that I have already deployed in Azure instead of the one provisioned by Azure Databricks?
asked 2024-06-05T08:40:13.1866667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(80, 52%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
Sujeet
0
Reputation points
commented 2024-06-12T05:04:01.3966667+00:00
PRADEEPCHEEKATLA-MSFT
83,886
Reputation points Microsoft Employee
1 answer
Integrating Databricks notebooks in Azure ML using SDK V2
Hi all, We currently have some Azure Databricks notebooks in production which we would like to integrate in Azure ML using the v2 SDK. I found resources to integrate these notebooks using the databricks_step in the v1 SDK. The official documentation…
asked 2024-06-07T13:12:55.3333333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(259.20000000000005, 18%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
Alexander
0
Reputation points
commented 2024-06-11T09:30:30.9966667+00:00
Amira Bedhiafi
18,341
Reputation points
1 answer
One of the answers was accepted by the question author.
Databricks Dev/Prod setup
We are a data team of 4 people. To make the process easy and more productive. Can we separate dev/prod environments at Databricks catalogue level rather than the workspace level? Can anyone share any thoughts on this? Thanks
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(150.4, 14.000000000000002%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EV%3C/text%3E%3C/svg%3E)
1 answer
One of the answers was accepted by the question author.
My Dev, test, prod environments are in different resource groups of same subscription. How do I create a devops pipeline in this case?a DevOps pipeline to deploy a
Hi, My dev, test and prod environments are in different resource groups of the same subscription. I am involved in a data engineering project where I will be using primarily below resources - ADLS - data storage ADF - Orchestration Azure Databricks - QC…
asked 2024-05-14T19:23:38.4366667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(70.4, 90%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EST%3C/text%3E%3C/svg%3E)
Shashwat Tiwary
40
Reputation points
accepted 2024-06-10T09:53:29.01+00:00
Shashwat Tiwary
40
Reputation points
2 answers
How to ignore the records in ADF Data Flows
Hi All I am building a data transamination using mapping data flows ,I have a time stamp field Like TimeStampUpdated in the target table. I want to lockup historical data with incremental data transamination and ignore the records coming in the…
asked 2024-05-23T06:58:12.53+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 6%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVR%3C/text%3E%3C/svg%3E)
venkat rao
65
Reputation points
commented 2024-06-10T08:32:00.3+00:00
PRADEEPCHEEKATLA-MSFT
83,886
Reputation points Microsoft Employee
2 answers
Access issue with app registration
I've created a Databricks workspace and a new notebook, but I don't have access to the secret keys under app registration, which are disabled for me. How can I solve this issue? Warning message You do not have access Your administrator has disabled the…
asked 2024-06-03T18:59:22.2666667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(278.4, 21%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENC%3C/text%3E%3C/svg%3E)
NIKHIL C
0
Reputation points
commented 2024-06-10T06:04:29.2033333+00:00
PRADEEPCHEEKATLA-MSFT
83,886
Reputation points Microsoft Employee
2 answers
Access to C:\Data not allowed . Error Code 22853
Access to C:\Data not allowed . Error Code 22853 Any workway around this ?
asked 2023-02-24T07:06:54.58+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(147.20000000000002, 46%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
Sushan
0
Reputation points
commented 2024-06-08T17:03:05.74+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(272, 45%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDD%3C/text%3E%3C/svg%3E)
Das, Dwaipayan
0
Reputation points
3 answers
How to specify a custom catalog name for Azure Databricks Delta Lake Dataset in ADF
Hello, I am creating an Azure Databricks Delta Lake Dataset in ADF and I am only able to choose the database name that links to Databricks's hive_metastore. How can I specify a custom catalog name that I created in Databricks instead of…
asked 2024-01-04T06:25:53.11+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(249.60000000000002, 17%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ETY%3C/text%3E%3C/svg%3E)
Tom Young
0
Reputation points
edited an answer 2024-06-07T13:17:10.84+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(294.40000000000003, 50%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EEL%3C/text%3E%3C/svg%3E)
Edward Loughran
0
Reputation points
1 answer
One of the answers was accepted by the question author.
How to Create Delta Table in Azure Synapse Analytics with Id Auto Increment Identity Column ?
I have created the Delta Lake Delta tables In ADLS using Synapse Notebook and in that table, I want to add an identity column (Auto increment 1,1) but I am not able to create the same, Below is my Create table script and error which i am facing. Table…
asked 2024-06-04T06:11:14.4766667+00:00
Vedant Desai
651
Reputation points
accepted 2024-06-07T09:52:14.62+00:00
Vedant Desai
651
Reputation points
1 answer
Databricks Spark Scala: RaiseError throws type error
Hello, I am facing an issue in Databricks 14.3-LTS within a Scala notebook. When I try to raise an Exception using Spark Catalyst with the following Scala code: import org.apache.spark.sql.types.{StringType, DateType} val errorMessage =…
asked 2024-06-05T08:24:16.67+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(9.6, 41%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EB%3C/text%3E%3C/svg%3E)
bn2302
0
Reputation points
commented 2024-06-07T04:19:55.0466667+00:00
PRADEEPCHEEKATLA-MSFT
83,886
Reputation points Microsoft Employee
2 answers
I am unable to mount containers using databricks and storage gen 2 ?
what is the issue?
asked 2024-06-05T15:57:33.72+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(51.2, 14.000000000000002%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESD%3C/text%3E%3C/svg%3E)
smriti das
0
Reputation points
answered 2024-06-06T12:19:54.16+00:00
Luis Arias
5,751
Reputation points
1 answer
One of the answers was accepted by the question author.
Can a single instance of Microsoft Purview scan multiple Azure Databricks Unity Catalog instances that exist in different logical data domains?
I am researching a use case where a single instance of Microsoft Purview can be used to scan multiple instances of Azure Databricks Unity Catalogs hosted in multiple logical / geographical domains, including using OpenLineage to provide lineage data to…
asked 2024-06-04T21:03:01.6766667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(105.60000000000001, 38%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EEH%3C/text%3E%3C/svg%3E)
Erich Huckschlag
20
Reputation points
accepted 2024-06-06T07:43:48.44+00:00
Erich Huckschlag
20
Reputation points
1 answer
Connecting Databricks to on prem sources
Is there a way to connect my Azure Databricks workspace to my local SQL Server database? I am trying to read data from my local SQL Server installed on my machine, but I am looking for a way to connect the two directly. I am aware we can use a SHIR with…
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,038 questions
Windows Network
Windows Network
Windows: A family of Microsoft operating systems that run across personal computers, tablets, laptops, phones, internet of things devices, self-contained mixed reality headsets, large collaboration screens, and other devices.Network: A group of devices that communicate either wirelessly or via a physical connection.
694 questions
asked 2024-05-29T11:03:25.18+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(220.8, 28.000000000000004%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAH%3C/text%3E%3C/svg%3E)
Abdullah Humayun
40
Reputation points
commented 2024-06-05T21:35:50.7366667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(86.4, 36%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBM%3C/text%3E%3C/svg%3E)
BhargavaGunnam-MSFT
28,446
Reputation points Microsoft Employee