load data from ADLS to Unity Catalog tables using Azure Data Factory?
What is the best way to load data from ADLS to Unity Catalog tables using Azure Data Factory?
Efficient Log Handling and Data Retention in Azure Data Factory and Databricks
I need to create a solution to send logs from Azure Data Factory to the Databricks Unity Catalog. I'm considering the following structure: Whenever an activity run results in either failure or success, the corresponding log will be sent to Azure Logic…
Creating a Zip File from Blob Storage Using Python in Azure Databricks
Hello, I am working on a task where I need to create a zip file for multiple files stored in blob storage, without having to read the files again or using local storage. I am using Python in Azure Databricks and would like to leverage its capabilities…
Azure Databricks Too Many Requests errors
We are getting many errors with loading notebooks and also now running jobs on clusters due to Databricks saying it has too many requests. For example, getting the below error message: run failed with error message Cluster '0724-103023-f2llqh3p' was…
Getting the size of parquet files from azure blob storage
I have a blob container abcd The folder structure is like below: abcd/Folder1/Folder a, Folder b…..Folder z Inside a particular Folder a/v1/full/20230505/part12344.parquet Similarly Folder b/v1/full/20230505/part9385795.parquet Scenario is I need to get…
Azure Databricks Billing
I am confused about how the databricks service is billed under Azure. From documentation, it is said that Databricks is totally integrated with Azure billing: one bill for both Azure infrastructure (VM, storage, Network traffic, etc) and Databricks…
Kafka Connector for Databricks
Can you forward me documentation to use Kafka Connector for Apache Spark Streaming job to connect to Azure Eventhub. I am looking for maven library version etc.
![](https://techprofile.blob.core.windows.net/images/UVyLn-Xlr0apNce5TRX1RA.png?8DBA3B)
Captures logs from Azure Data Factory and inserts them into a Delta Table in Databricks
Good morning, I need assistance in creating a project that captures logs from Azure Data Factory and inserts them into a Delta Table in Databricks. The key requirements for this project are as follows: No Duplicate Logs: Ensuring that the logs are not…
how to fetch data from Azure Active Directory(AD) by using either ADF or databricks
To fetch data from Azure Active Directory (AD) using either Azure Data Factory (ADF) or Azure Databricks, Pleae let me know in detail. thanks
Data Factory monitoring by inserting data table
Hello, I would like to know the best way to insert Datafactory activity logs into my Databricks delta table, so that I can use dashbosrd and create monitoring in Databricks itself , can you help me? I would like every 5 minutes for all activity logs in…
How to use databricks ai to auto generate data definitions for all the tables in my database?
I know we can go to the catalog in databricks and generate data definitions for columns inside of our database using ai, but is there a way of automatically generating these definitions without have to manually generate them and click accept on every…
SAS token generation by Databricks to access CSV files from ADLS container folder
Hi Team, There are some csv files zips inside the ADLS container folder. These zip files need to be downloaded for data correction. Downloading the file requires SAS token embedded with zip file path. Databricks has been used to generate the token and…
Azure Data Bricks - User Doesn't have permission to perform this action while connecting to Azure Synapse Dedicate Pool
We are connecting Azure Synapse Analytics - Dedicated Pool using the PySpark Code that runs from Azure Data Bricks using SQL Authentication. While running, we are getting the below error when we use a user with db_datawriter and db_datareader…
Azure Databricks - Timeout error after 60 minutes when launching an Azure Databricks cluster
When I attempt to start a cluster through the Azure Databricks portal/UI, after 30 minutes I receive the following error in the event log: Failed to add 3 containers to the compute. Will attempt retry: true. Reason: Cloud provider launch failure Azure…
![](https://techprofile.blob.core.windows.net/images/VfQFAmOikEWfBHko2XlWTA.png?8D7F33)
Append in Liquid Cluster enabled table is not completing on DBR 15.3 version
I am trying do analysis with a Partition Table and Liquid Clustered table. As per Azure Databricks recommendation, I am using DBR 15.2 to execute the code. I have created a clustered table as and using an append operation which is specified below. Few…
![](https://techprofile.blob.core.windows.net/images/VfQFAmOikEWfBHko2XlWTA.png?8D7F33)
How to add parent group for one specific group in databricks?
The question for this is in "databricks" -> "settings" -> "Identity and access" -> "Groups" Here has "admin" group with system managed. But we wonder if a new group can be assigned as a parent…
My python code uses AzureCliCredential() function, but it is giving error in running in Synapse/ADF/Databricks notebook .
My python code uses AzureCliCredential() function, but it is giving error in running in Synapse Workspace that your Azure Cli command not found on path . I have tried using other function like DefaultAzureCredentials(), ClientSecretCredentials () also…
Workflow that logs the completion of certain pipelines into a table
I'm having a lot of difficulty implementing the solutions I had in mind. Previously, I asked for help to create an architecture that would be efficient, easy to maintain, and cost-effective. I received various suggestions, but I can't decide which one to…
![](https://techprofile.blob.core.windows.net/images/UVyLn-Xlr0apNce5TRX1RA.png?8DBA3B)
Datafactory logs for dasboard on databricks
hello! need help! I need to solve a problem that consists of taking completed pipeline logs with specific names and inserting them into a deltatable within Databricks to create dashboards in it. this solution needs to be entirely using Azure tools. I'm…
data factory azure integration runtime with private end points
I want to create data factory with private end points, Can I use Azure integration runtime with private end points? Is it possible to use Azure integration runtime with private end points? Or "self hosted" is ONLY possible option…