Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
2,532 questions
0 answers
Connect to Blob storage from Azure Databricks SQL
So I would like to read a table from a CSV file on Azure Blob Storage in my own account, and load it into a table in Unity Catalog on databricks (hopefully using SQL). I have tried this SQL command: CREATE TABLE IF NOT EXISTS <table_name>; COPY…
asked 2024-06-11T13:36:31.3633333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(217.60000000000002, 47%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKW%3C/text%3E%3C/svg%3E)
Kaizad Wadia
0
Reputation points
commented 2024-06-11T13:53:57.4566667+00:00
Vinodh247-1375
12,046
Reputation points
1 answer
Integrating Databricks notebooks in Azure ML using SDK V2
Hi all, We currently have some Azure Databricks notebooks in production which we would like to integrate in Azure ML using the v2 SDK. I found resources to integrate these notebooks using the databricks_step in the v1 SDK. The official documentation…
asked 2024-06-07T13:12:55.3333333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(259.20000000000005, 18%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
Alexander
0
Reputation points
commented 2024-06-11T09:30:30.9966667+00:00
Amira Bedhiafi
17,706
Reputation points
1 answer
One of the answers was accepted by the question author.
Databricks Dev/Prod setup
We are a data team of 4 people. To make the process easy and more productive. Can we separate dev/prod environments at Databricks catalogue level rather than the workspace level? Can anyone share any thoughts on this? Thanks
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(150.4, 14.000000000000002%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EV%3C/text%3E%3C/svg%3E)
2 answers
When creating a second external location to the same path in Azure Databricks Unity Catalog it gives conflicting error for path. Is there any way to solve this?
Hello Team, When creating a second external location/external volumes to the same path with different folder or to the root location gives an error see below for details in Azure Databricks Unity Catalog as it gives conflict error for path. Is there any…
asked 2024-06-04T10:23:22.8733333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 24%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAG%3C/text%3E%3C/svg%3E)
Ashwini Gaikwad
85
Reputation points
commented 2024-06-10T13:53:17.2833333+00:00
Ashwini Gaikwad
85
Reputation points
1 answer
org.apache.hadoop.fs.FileAlreadyExistsException: Failed to rename temp file
[Repeat Question due to old thread] We have built a streaming pipeline with spark autoloader. Source Folder is a azure blob container. We've encountered a rare issue (could not replicate it). Below is the exception…
asked 2024-06-07T01:40:56.47+00:00
Hiran Amarathunga
5
Reputation points
commented 2024-06-10T13:19:32.7533333+00:00
Hiran Amarathunga
5
Reputation points
1 answer
Issues while writing into bad_records path in Databricks
Hello All, I would like to get your inputs with a scenario that I see while writing into the bad_records file. I am reading a ‘Ԓ’ delimited CSV file based on a schema that I have already defined. I have enabled error handling while reading the file to…
asked 2024-06-05T01:13:41.47+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(38.4, 78%, 13%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAT%3C/text%3E%3C/svg%3E)
Alok Thampi
91
Reputation points
answered 2024-06-10T12:16:55.65+00:00
Alok Thampi
91
Reputation points
1 answer
One of the answers was accepted by the question author.
My Dev, test, prod environments are in different resource groups of same subscription. How do I create a devops pipeline in this case?a DevOps pipeline to deploy a
Hi, My dev, test and prod environments are in different resource groups of the same subscription. I am involved in a data engineering project where I will be using primarily below resources - ADLS - data storage ADF - Orchestration Azure Databricks - QC…
asked 2024-05-14T19:23:38.4366667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(70.4, 90%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EST%3C/text%3E%3C/svg%3E)
Shashwat Tiwary
20
Reputation points
accepted 2024-06-10T09:53:29.01+00:00
Shashwat Tiwary
20
Reputation points
1 answer
The scim API is by default adding users to admins group in azure databricks
Hi, When we are invoking scim API in azure databricks it is by default adding users to the admins group and also after deleting users from only admins group they are being created again. Also calling scim API with adding groups as users also adding them…
asked 2024-06-05T11:29:19.43+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(262.40000000000003, 26%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGN%3C/text%3E%3C/svg%3E)
Gupta, Neha
0
Reputation points
commented 2024-06-10T09:23:17.1033333+00:00
Gupta, Neha
0
Reputation points
1 answer
Azure Data Bricks - User Doesn't have permission to perform this action while connecting to Azure Synapse Dedicate Pool
We are connecting Azure Synapse Analytics - Dedicated Pool using the PySpark Code that runs from Azure Data Bricks using SQL Authentication. While running, we are getting the below error when we use a user with db_datawriter and db_datareader…
asked 2024-06-07T11:26:00.4933333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(230.39999999999998, 97%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPS%3C/text%3E%3C/svg%3E)
Praveen Sreeram
1
Reputation point
commented 2024-06-10T09:16:26.0533333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 8%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EST%3C/text%3E%3C/svg%3E)
Smaran Thoomu
11,290
Reputation points Microsoft Vendor
2 answers
How to ignore the records in ADF Data Flows
Hi All I am building a data transamination using mapping data flows ,I have a time stamp field Like TimeStampUpdated in the target table. I want to lockup historical data with incremental data transamination and ignore the records coming in the…
asked 2024-05-23T06:58:12.53+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 6%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVR%3C/text%3E%3C/svg%3E)
venkat rao
65
Reputation points
commented 2024-06-10T08:32:00.3+00:00
PRADEEPCHEEKATLA-MSFT
81,646
Reputation points Microsoft Employee
1 answer
How can i connect Azure Databricks to Neo4j??
Hello, I want to connect to neo4j from Azure Databricks. What are the different approaches do I have? I am trying to connect here and i getting following error. Do I need to do anything before running the code? i mean setup managed identity or enable…
asked 2024-06-10T03:41:31.9866667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(201.6, 73%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESR%3C/text%3E%3C/svg%3E)
Siddartha Reddy Jammula
20
Reputation points
answered 2024-06-10T07:23:32.1533333+00:00
PRADEEPCHEEKATLA-MSFT
81,646
Reputation points Microsoft Employee
1 answer
"Premium Automated Serverless Compute - Promo DBU" expenses arise from what, how can I disable it, and why are the costs so high?
"Premium Automated Serverless Compute - Promo DBU" expenses arise from what, how can I disable it, and why are the costs so high? detail in below
asked 2024-06-09T16:43:36.0033333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(70.4, 0%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPT%3C/text%3E%3C/svg%3E)
Pratya Thanwatthanakit
0
Reputation points
edited an answer 2024-06-10T06:43:03.4633333+00:00
PRADEEPCHEEKATLA-MSFT
81,646
Reputation points Microsoft Employee
0 answers
Databricks Simba Spark ODBC .NET8 C# Driver Parameters in SQL Queries
Hello, I'm using Simba ODBC driver v2.8.0 in order to query data from my azure databrick sql warehouse into a .net 8 Asp.net Api App. The ODBC driver works fine using plain text query but i need to parametrize the query. Searching around I found that it…
asked 2024-06-03T15:30:12.4666667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(265.6, 63%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ELN%3C/text%3E%3C/svg%3E)
Luigi Navarra
5
Reputation points
commented 2024-06-10T06:05:14.48+00:00
PRADEEPCHEEKATLA-MSFT
81,646
Reputation points Microsoft Employee
2 answers
Access issue with app registration
I've created a Databricks workspace and a new notebook, but I don't have access to the secret keys under app registration, which are disabled for me. How can I solve this issue? Warning message You do not have access Your administrator has disabled the…
asked 2024-06-03T18:59:22.2666667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(278.4, 21%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENC%3C/text%3E%3C/svg%3E)
NIKHIL C
0
Reputation points
commented 2024-06-10T06:04:29.2033333+00:00
PRADEEPCHEEKATLA-MSFT
81,646
Reputation points Microsoft Employee
0 answers
Azure Databricks exercise error
Keep receiving the error "No such file or directory /your_correct_source_value/wikipedia/pagecounts/staging_parquet_en_only_clean" When I checked Wikipedia, it appears this dataset has been deprecated since 2016-08-01 Could a new dataset be…
asked 2024-06-05T19:59:42.4166667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(291.2, 46%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJO%3C/text%3E%3C/svg%3E)
Joab Odera
0
Reputation points
commented 2024-06-10T05:36:19.4366667+00:00
PRADEEPCHEEKATLA-MSFT
81,646
Reputation points Microsoft Employee
1 answer
deploying Azure databrick with datalake
Deploying Azure Databricks creates an additional resource group in the background, which includes a data lake. Is it possible to use the data lake that I have already deployed in Azure instead of the one provisioned by Azure Databricks?
asked 2024-06-05T08:40:13.1866667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(80, 52%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
Sujeet
0
Reputation points
commented 2024-06-10T05:32:53.85+00:00
PRADEEPCHEEKATLA-MSFT
81,646
Reputation points Microsoft Employee
2 answers
Access to C:\Data not allowed . Error Code 22853
Access to C:\Data not allowed . Error Code 22853 Any workway around this ?
asked 2023-02-24T07:06:54.58+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(147.20000000000002, 46%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
Sushan
0
Reputation points
commented 2024-06-08T17:03:05.74+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(272, 45%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDD%3C/text%3E%3C/svg%3E)
Das, Dwaipayan
0
Reputation points
3 answers
How to specify a custom catalog name for Azure Databricks Delta Lake Dataset in ADF
Hello, I am creating an Azure Databricks Delta Lake Dataset in ADF and I am only able to choose the database name that links to Databricks's hive_metastore. How can I specify a custom catalog name that I created in Databricks instead of…
asked 2024-01-04T06:25:53.11+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(249.60000000000002, 17%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ETY%3C/text%3E%3C/svg%3E)
Tom Young
0
Reputation points
edited an answer 2024-06-07T13:17:10.84+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(294.40000000000003, 50%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EEL%3C/text%3E%3C/svg%3E)
Edward Loughran
0
Reputation points
1 answer
One of the answers was accepted by the question author.
How to Create Delta Table in Azure Synapse Analytics with Id Auto Increment Identity Column ?
I have created the Delta Lake Delta tables In ADLS using Synapse Notebook and in that table, I want to add an identity column (Auto increment 1,1) but I am not able to create the same, Below is my Create table script and error which i am facing. Table…
asked 2024-06-04T06:11:14.4766667+00:00
Vedant Desai
651
Reputation points
accepted 2024-06-07T09:52:14.62+00:00
Vedant Desai
651
Reputation points
0 answers
Restricting files/folders to upload into External volumes in Azure databricks UC workspace
Hello Team, Is there a way to restrict the files or folders to upload/download from external volumes same like DBFS? Is there any option to disable the uploading files/folders feature in external volumes of azure databricks workspace with Unity Catalog.…
asked 2024-06-06T17:37:57.5833333+00:00
Ashwini Gaikwad
85
Reputation points
commented 2024-06-07T07:57:14.7166667+00:00
Ashwini Gaikwad
85
Reputation points