Cluster Not Created in Pay as you go subscription
I have a pay as you subscription, I have created my databricks workspace in US East Region. Whatever i cluster i created with single or multi node. every time i am facing Quota limit exceeded error. What is the best available cluster in US east region,…
I want to give a static public IP address to my Databricks clusters
I could go over the "private" Microsoft network backbone using something like ExpressRoute. Connecting to a firewall outside of Databricks. But for now I just need to connect over the public internet to build a proof of concept. Would need to…
transformation changes in silver and bronze layer
Hi , what transformation takes place between silver and gold layer .i.e have loaded data in bromze layer whichn is my bronze layer and i transformed data here ... then what will happend to silver to gold layer apart from pk,fk joins and all
CORS Issues Between Azure Static Web Apps and Azure Databricks
Hi, I'm currently facing some CORS issues in my application setup and would like to know your opinion on how to solve them. Front-end: Angular application deployed on Azure Static Web Apps (.azurestaticapps.net) Endpoint to access: Model serving…
What is the best way to access data in the data bricks by using azure function?
I just tried to load data from data bricks by using data bricks jobs API and azure function. Can I know is there another way to do the same thing that based on azure function?
Azure Databricks Too Many Requests errors
We are getting many errors with loading notebooks and also now running jobs on clusters due to Databricks saying it has too many requests. For example, getting the below error message: run failed with error message Cluster '0724-103023-f2llqh3p' was…
load data from ADLS to Unity Catalog tables using Azure Data Factory?
What is the best way to load data from ADLS to Unity Catalog tables using Azure Data Factory?
Efficient Log Handling and Data Retention in Azure Data Factory and Databricks
I need to create a solution to send logs from Azure Data Factory to the Databricks Unity Catalog. I'm considering the following structure: Whenever an activity run results in either failure or success, the corresponding log will be sent to Azure Logic…
Creating a Zip File from Blob Storage Using Python in Azure Databricks
Hello, I am working on a task where I need to create a zip file for multiple files stored in blob storage, without having to read the files again or using local storage. I am using Python in Azure Databricks and would like to leverage its capabilities…
Getting the size of parquet files from azure blob storage
I have a blob container abcd The folder structure is like below: abcd/Folder1/Folder a, Folder b…..Folder z Inside a particular Folder a/v1/full/20230505/part12344.parquet Similarly Folder b/v1/full/20230505/part9385795.parquet Scenario is I need to get…
Azure Databricks Billing
I am confused about how the databricks service is billed under Azure. From documentation, it is said that Databricks is totally integrated with Azure billing: one bill for both Azure infrastructure (VM, storage, Network traffic, etc) and Databricks…
Kafka Connector for Databricks
Can you forward me documentation to use Kafka Connector for Apache Spark Streaming job to connect to Azure Eventhub. I am looking for maven library version etc.
Captures logs from Azure Data Factory and inserts them into a Delta Table in Databricks
Good morning, I need assistance in creating a project that captures logs from Azure Data Factory and inserts them into a Delta Table in Databricks. The key requirements for this project are as follows: No Duplicate Logs: Ensuring that the logs are not…
how to fetch data from Azure Active Directory(AD) by using either ADF or databricks
To fetch data from Azure Active Directory (AD) using either Azure Data Factory (ADF) or Azure Databricks, Pleae let me know in detail. thanks
Data Factory monitoring by inserting data table
Hello, I would like to know the best way to insert Datafactory activity logs into my Databricks delta table, so that I can use dashbosrd and create monitoring in Databricks itself , can you help me? I would like every 5 minutes for all activity logs in…
How to use databricks ai to auto generate data definitions for all the tables in my database?
I know we can go to the catalog in databricks and generate data definitions for columns inside of our database using ai, but is there a way of automatically generating these definitions without have to manually generate them and click accept on every…
SAS token generation by Databricks to access CSV files from ADLS container folder
Hi Team, There are some csv files zips inside the ADLS container folder. These zip files need to be downloaded for data correction. Downloading the file requires SAS token embedded with zip file path. Databricks has been used to generate the token and…
Azure Data Bricks - User Doesn't have permission to perform this action while connecting to Azure Synapse Dedicate Pool
We are connecting Azure Synapse Analytics - Dedicated Pool using the PySpark Code that runs from Azure Data Bricks using SQL Authentication. While running, we are getting the below error when we use a user with db_datawriter and db_datareader…
Azure Databricks - Timeout error after 60 minutes when launching an Azure Databricks cluster
When I attempt to start a cluster through the Azure Databricks portal/UI, after 30 minutes I receive the following error in the event log: Failed to add 3 containers to the compute. Will attempt retry: true. Reason: Cloud provider launch failure Azure…
Append in Liquid Cluster enabled table is not completing on DBR 15.3 version
I am trying do analysis with a Partition Table and Liquid Clustered table. As per Azure Databricks recommendation, I am using DBR 15.2 to execute the code. I have created a clustered table as and using an append operation which is specified below. Few…