Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,938 questions
0 answers
Why create compute is taking long time?
I am trying create a compute for my workspaces i tried every combination still it is not working
asked 2024-05-03T13:41:31+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(300.8, 13%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAP%3C/text%3E%3C/svg%3E)
Aditya Parida
0
Reputation points
asked 2024-05-03T13:41:31+00:00
Aditya Parida
0
Reputation points
1 answer
Rest API call towards Azure Storage account (ADLSgen2) from Databricks (write to Delta) fails when Authentication is via ACL and SP
In the nutshell. I work in very sensitive and security intense environment. We have decided to use ACL (instead of RBAC/ABAC) for authorization to achieve finer control over Storage account. For our Databricks service, we use only job cluster and job is…
Azure Storage Accounts
Azure Storage Accounts
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
2,714 questions
Azure
Azure
A cloud computing platform and infrastructure for building, deploying and managing applications and services through a worldwide network of Microsoft-managed datacenters.
969 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,938 questions
asked 2024-05-01T14:51:16.1566667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(128, 92%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESO%3C/text%3E%3C/svg%3E)
Senkyr, Oldrich
0
Reputation points
edited an answer 2024-05-03T11:03:17.1533333+00:00
Senkyr, Oldrich
0
Reputation points
1 answer
How to configure ADF pipeline run, linked service, so it uses Databricks serverless compute
Databricks has recently announced serverless compute for workflows: https://learn.microsoft.com/en-us/azure/databricks/workflows/jobs/run-serverless-jobs I would like to be able to execute Azure Data Factory (ADF) jobs using this…
asked 2024-05-01T12:12:06.9033333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 31%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKP%3C/text%3E%3C/svg%3E)
Krzysztof Przysowa
0
Reputation points
edited a comment 2024-05-03T09:47:03.68+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 61%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EP%3C/text%3E%3C/svg%3E)
phemanth
5,840
Reputation points Microsoft Vendor
0 answers
How to parse nested json array of document in ADF data flow
Hi all I am trying to fitch the values from a nested josh array of document , I have used aggregate to convert into objects but not able to fitch the values of all child nodes like as below itOffer.item itOffer.item.SplOfr itOffer.item.buy …
asked 2024-05-02T15:44:57.8633333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 6%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVR%3C/text%3E%3C/svg%3E)
venkat rao
45
Reputation points
edited a comment 2024-05-03T08:34:54.1733333+00:00
phemanth
5,840
Reputation points Microsoft Vendor
1 answer
Data size of databricks delta tables
It has been observed that the size of delta tables are much less as compared to when checked the underlying delta files in the storage account. Suppose a databricks delta table raw.deltaTableA has size of 2MB if we check the size of underlying delta…
asked 2024-05-02T09:39:01.4133333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(230.39999999999998, 23%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENK%3C/text%3E%3C/svg%3E)
NIKHIL KUMAR
81
Reputation points
commented 2024-05-03T05:22:36.3033333+00:00
Vinodh247-1375
11,211
Reputation points
2 answers
One of the answers was accepted by the question author.
Databricks to Table storage Data load
Hi Team, Currently, I have data bricks spark jobs running which load data from Blob Storage and then process it using Databricks and then dump the clean data into another blob storage. Now, I would like to dump the cleaned file to Table storage…
asked 2021-03-31T09:36:03.387+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(54.400000000000006, 55.00000000000001%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EIM%3C/text%3E%3C/svg%3E)
Imran Mondal
246
Reputation points
commented 2024-05-03T04:02:14.3633333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(96, 86%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERS%3C/text%3E%3C/svg%3E)
Rubal Sharma
0
Reputation points
1 answer
Databricks support redirects to azure support: unexpected internal error when spinning up a Databricks all-purpose cluster
Hello, What do we do when we get this error, when spinning up a Databricks all-purpose cluster? { "reason": { "code": "CONTAINER_LAUNCH_FAILURE", "type": "SERVICE_FAULT", …
asked 2024-05-02T13:44:55.72+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(240, 18%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAS%3C/text%3E%3C/svg%3E)
ADM.Susana Domingos
0
Reputation points
answered 2024-05-02T20:49:11.4233333+00:00
Amira Bedhiafi
15,676
Reputation points
1 answer
Failure on Write EventSubscription - Internal error
I am trying to set up Databricks Autoloader with File Notifications. Every time I get a failure on the EventSubscription/write operation. I have tried giving the relevant account as much access as I can but still nothing. { "statusMessage":…
asked 2024-04-26T21:07:19.1333333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(179.20000000000002, 38%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBJ%3C/text%3E%3C/svg%3E)
Bradley Jamrozik
0
Reputation points
commented 2024-05-02T12:41:03.4366667+00:00
Bradley Jamrozik
0
Reputation points
1 answer
Error with Create Table USING DELTA LOCATION in training exercise
In the exercise https://microsoftlearning.github.io/mslearn-databricks/Instructions/Exercises/03-Delta-lake-in-Azure-Databricks.html the line of code spark.sql("CREATE TABLE AdventureWorks.ProductsExternal USING DELTA LOCATION…
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,938 questions
Azure Training
Azure Training
Azure: A cloud computing platform and infrastructure for building, deploying and managing applications and services through a worldwide network of Microsoft-managed datacenters.Training: Instruction to develop new skills.
942 questions
asked 2024-05-01T13:00:09.32+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(6.4, 31%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJM%3C/text%3E%3C/svg%3E)
James Mitchell
0
Reputation points
commented 2024-05-02T12:14:59.4633333+00:00
James Mitchell
0
Reputation points
0 answers
Azure Databricks workflow job failure
We have a stream workflow job that run 24*7 and loads the data in delta table for say: raw.deltaTableA Now, the problem is in case we are trying to optimize this delta (optimize raw.deltaTableA) table while the table is getting loaded we get frequent…
asked 2024-05-02T08:23:36.4266667+00:00
NIKHIL KUMAR
81
Reputation points
commented 2024-05-02T10:21:25.59+00:00
PRADEEPCHEEKATLA-MSFT
77,751
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Do you know how to install the 'ODBC Driver 17 for PostgreSQL' on a Azure Databricks cluster?
I am attempting to run postgreSQL stored procedures , through Azure Databricks notebook. We have stored procedure written in Azure Database for PostgreSQL and we want to run postgreSQL stored procedures through Azure Databricks Notebook (using…
asked 2024-04-30T11:06:56.06+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(96, 57.99999999999999%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAS%3C/text%3E%3C/svg%3E)
Anuj, Singh (Cognizant)
25
Reputation points
accepted 2024-05-02T09:34:32.8666667+00:00
Anuj, Singh (Cognizant)
25
Reputation points
asked 2024-05-01T06:57:01.2766667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(179.20000000000002, 53%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
Alpha
0
Reputation points
commented 2024-05-02T09:22:50.38+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(297.6, 81%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAS%3C/text%3E%3C/svg%3E)
Amrinder Singh
2,195
Reputation points Microsoft Employee
0 answers
DatabricksSQL Logs and correlate with query history
Hi everyone, I'm currently working on capturing logging information about query executions and data downloads within a Databricks workspace. Here's a summary of my current setup and the issue I'm facing: Diagnostic Settings in Azure Databricks: I have…
asked 2024-04-22T11:09:19.4+00:00
Julio Avellaneda
0
Reputation points
commented 2024-05-02T03:56:38.2633333+00:00
PRADEEPCHEEKATLA-MSFT
77,751
Reputation points Microsoft Employee
1 answer
How create a unity catalogue for existing Azure Databricks workspaces with all permissions as identity governance administrator?
Trying to create a unity catalogue for easy flow of machine learning solution on Databricks which has 3 separate workspaces - development, staging and production.
asked 2024-04-24T03:22:24.2566667+00:00
Mohd Zubair Humza
0
Reputation points
commented 2024-05-02T03:45:50.16+00:00
PRADEEPCHEEKATLA-MSFT
77,751
Reputation points Microsoft Employee
1 answer
Databricks OutOfMemory error on code that previously worked without issue
I have a notebook in Azure Databricks that converts a list of columns in a bronze tier table into individual child rows in a silver tier table. This notebook was previously running (for weeks) without issue. Suddenly, I am now consistently receiving an…
asked 2024-04-26T22:03:44.67+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 98%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESM%3C/text%3E%3C/svg%3E)
Shane McGarry
0
Reputation points
commented 2024-05-02T03:36:20.2733333+00:00
PRADEEPCHEEKATLA-MSFT
77,751
Reputation points Microsoft Employee
0 answers
Databricks OutOfMemory error on code that previously worked without issue
I have a notebook in Azure Databricks that does some transformations on a bronze tier table and inserts the transformed data into a silver tier table. This notebook is used to do an initial load of the data from our existing system into our new datalake.…
asked 2024-04-26T22:23:15.07+00:00
Shane McGarry
0
Reputation points
commented 2024-05-02T03:35:59.5333333+00:00
PRADEEPCHEEKATLA-MSFT
77,751
Reputation points Microsoft Employee
1 answer
How to ship Azure Databricks artifacts from Dev->QA->Prod through Azure Devops Pipelines?
We have a Azure Databricks workspace and Dev/QA/Prod environments. Everytime the Data engineers have to ship the artifacts from nonprod -> prod (e.g. python notebooks, config modules, etc) they have to copy the artifacts manually over to the next…
asked 2024-04-29T21:17:03.8233333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 89%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EC%3C/text%3E%3C/svg%3E)
Cataster
641
Reputation points
commented 2024-05-02T03:30:34.6966667+00:00
PRADEEPCHEEKATLA-MSFT
77,751
Reputation points Microsoft Employee
1 answer
How do I add an inbound security rule if there is an default DenyAllInbound Rule that causes an error when attempting to create an inbound rule?
|Received an email with: The public IP address range for the Azure Databricks control plane will be updated on 30 May 2024—you may need to take action You're receiving this email because you use Azure Databricks. To support infrastructure …
asked 2024-04-30T17:39:32.7466667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(64, 18%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPS%3C/text%3E%3C/svg%3E)
Parris Sikorski (ALLEGIS GROUP HOLDINGS INC)
0
Reputation points Microsoft Vendor
commented 2024-05-02T03:26:05.1766667+00:00
PRADEEPCHEEKATLA-MSFT
77,751
Reputation points Microsoft Employee
0 answers
Custom libraries (wheel) for ADF Databricks Python activity run on serverless compute
I want to be able to execute Python scripts (via Databricks Python) from Azure Data Factory using serverless compute. Serverless compute does not support cluster level (compute scoped) libraries. In databricks workflows, it is being done as…
asked 2024-05-01T12:30:52.2366667+00:00
Krzysztof Przysowa
0
Reputation points
commented 2024-05-02T02:34:43.2433333+00:00
PRADEEPCHEEKATLA-MSFT
77,751
Reputation points Microsoft Employee
0 answers
How do I use the Script activity in ADF, so it uses Azure Databricks SQL Warehouse
I want to be able to use ADF Script activity to execute SQL statements on the Azure Databricks SQL warehouses (including the serverless kind). https://learn.microsoft.com/en-us/azure/data-factory/transform-data-using-script Azure Databricks SQL…
asked 2024-05-01T12:21:53.01+00:00
Krzysztof Przysowa
0
Reputation points
commented 2024-05-01T20:08:24.46+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(86.4, 36%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBM%3C/text%3E%3C/svg%3E)
BhargavaGunnam-MSFT
26,306
Reputation points Microsoft Employee