2,534 questions with Azure Databricks tags

Sort by: Updated
1 answer One of the answers was accepted by the question author.

Azure Data Bricks Cluster

Hi, While creating Azure Databricks cluster I am getting an error that the node Standrad_DS3_vs is not available in the Region. Please help. I am not able to see any Region while I run a Azure bash command to find the Region which has Standard_DS3_v2…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,534 questions
asked 2025-07-07T17:25:51.37+00:00
Rishi Rithi 0 Reputation points
accepted 2025-07-08T16:12:56.2566667+00:00
Rishi Rithi 0 Reputation points
2 answers One of the answers was accepted by the question author.

How to implement retry logic in Azure Databricks for a failed microbatch in Structured Streaming (without replaying from Kafka)?

We are ingesting CDC data from Kafka into Azure Databricks using Structured Streaming. In rare cases, a microbatch fails due to reconciliation logic (e.g., row count or hash mismatch). We would like to reprocess only the failed batch (e.g., write the…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,534 questions
asked 2025-07-07T09:31:20.03+00:00
Gabriel25 545 Reputation points
accepted 2025-07-08T11:34:11.8866667+00:00
Gabriel25 545 Reputation points
1 answer

how to connect onprem hadoop hive to azure databricks

how to connect onprem hadoop hive to azure databricks

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,534 questions
asked 2025-07-03T19:27:41.45+00:00
Prasad Sandu 0 Reputation points
commented 2025-07-08T11:33:08.3766667+00:00
Smaran Thoomu 24,575 Reputation points Microsoft External Staff Moderator
1 answer

Unable to access account console

We are currently unable to access the Databricks Account Console (https://accounts.azuredatabricks.net) for our Azure Databricks deployment. Attempts to log in using our Azure AD Global Administrator account redirect us to the workspace instead of the…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,534 questions
asked 2025-07-04T23:51:53.9833333+00:00
Chaithanya Chowdhary 0 Reputation points
commented 2025-07-08T11:19:13.25+00:00
Smaran Thoomu 24,575 Reputation points Microsoft External Staff Moderator
0 answers

Not able to open the databricks work space(adb-1457270792635633.13.azuredatabricks.net)

I am not able to open the azure data bricks work space (adb-1457270792635633.13.azuredatabricks.net). Need your help on it pls .

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,534 questions
asked 2025-07-07T04:19:18.24+00:00
Yakkli, Bala Ramesh 0 Reputation points
commented 2025-07-08T11:11:56.1266667+00:00
Smaran Thoomu 24,575 Reputation points Microsoft External Staff Moderator
1 answer One of the answers was accepted by the question author.

How to set up databricks-bundle in Azure Databricks UI

Hello Mr./Ms., I'm a new bee with Databircks and databricks-bundle for CICD. I have 3 questions:. By chance, I saw this picture: I tried to look for documents or videos related to Databricks Bundles in the UI (like in the image above), including how…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,534 questions
asked 2025-07-07T14:35:28.33+00:00
Viet Tran 50 Reputation points
commented 2025-07-08T01:45:47.8966667+00:00
Viet Tran 50 Reputation points
2 answers One of the answers was accepted by the question author.

Best practices for secure and performant JDBC access from Azure Databricks to on-prem Hadoop Hive over VPN

We have a VNet-injected Azure Databricks workspace connected to on-prem Hadoop via VPN Gateway. We want to read from Hive tables using the JDBC driver in Databricks notebooks. While the connection works, performance is slow and we're unsure about the…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,534 questions
asked 2025-07-07T10:28:55.65+00:00
Vikranth-2626 160 Reputation points
accepted 2025-07-07T14:59:09.1233333+00:00
Vikranth-2626 160 Reputation points
1 answer

How to selectively re-run only failed ETL phases (extract/transform/load) for specific partitions in a continuous pipeline?

Question: In our current reconciliation framework, we log the success/failure of each ETL pipeline phase—extraction, transformation, and load—using dedicated status flags (extraction_status, transformation_status, load_status) in a phase-level status…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,534 questions
asked 2025-07-03T08:09:15.2066667+00:00
Janice Chi 160 Reputation points
commented 2025-07-07T06:49:16.4033333+00:00
Smaran Thoomu 24,575 Reputation points Microsoft External Staff Moderator
1 answer

How to disable serverless culster in Azure Databricks?

How to disable serverless culster in Azure Databricks? Mail to Suport team asking paid subscription

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,534 questions
asked 2025-07-06T05:16:08.8466667+00:00
Akash Nidavani 0 Reputation points
commented 2025-07-07T06:36:48.0266667+00:00
Smaran Thoomu 24,575 Reputation points Microsoft External Staff Moderator
1 answer

Best Retry Strategy for Reconciliation-Failed Microbatches in Real-Time CDC Pipelines (Azure SQL + Databricks)

We are implementing a near-real-time CDC streaming pipeline using Azure Databricks and Azure SQL Hyperscale. Flattened CDC data from Kafka is ingested into Delta Lake (ADLS) per microbatch, and each batch is also written to a staging table in Azure SQL.…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,534 questions
asked 2025-07-01T17:40:01.7866667+00:00
Janice Chi 160 Reputation points
commented 2025-07-07T06:29:49.61+00:00
Smaran Thoomu 24,575 Reputation points Microsoft External Staff Moderator
1 answer

Databricks Execution Strategy for One-Time 80 TB Migration

In our one-time historical data migration project, we are processing approximately 80 TB of CSV data into Parquet format using Azure Databricks, within a strict 3-day completion window. The raw CSV files are extracted from on-prem DB2 snapshots and…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,534 questions
asked 2025-07-05T12:16:38.5433333+00:00
Janice Chi 160 Reputation points
commented 2025-07-07T05:19:41.85+00:00
Smaran Thoomu 24,575 Reputation points Microsoft External Staff Moderator
0 answers

The workspace is in a failed state and cannot be launched.

When I try to delete a Databricks resource - I get the error... The workspace 'workspace_name' is in a failed state and cannot be launched. Please review error details in the activity log tab and retry your operation. How do I resolve this? I've…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,534 questions
asked 2025-07-02T16:16:31.1133333+00:00
Sukhminder Bining 0 Reputation points
commented 2025-07-05T14:29:32.1+00:00
Sukhminder Bining 0 Reputation points
0 answers

Azure Databricks pytorch error

Though i have check Machine learning checkbox. When i try to run BartForConditionalGeneration it shows Pytorch not installed. I saw the support page they have mentioned to install pytorch for CPU based clusters with wheel but where and how should i run…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,534 questions
asked 2025-07-03T14:57:04.44+00:00
Aryan Rastogi 60 Reputation points
commented 2025-07-04T14:17:06.9066667+00:00
Venkat Reddy Navari 3,470 Reputation points Microsoft External Staff Moderator
1 answer

Using Dynamic Job Parameters with Databricks Job Activity

A replacement is being implemented for a generic Databricks job trigger pipeline with the new Databricks job activity. The goal is to maintain all job triggering logic in a separate pipeline. Currently, a JSON-like string is being dynamically passed with…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,534 questions
asked 2025-06-30T08:25:18.73+00:00
Víctor C. Fernández 0 Reputation points
commented 2025-07-04T14:12:45.2433333+00:00
Venkat Reddy Navari 3,470 Reputation points Microsoft External Staff Moderator
2 answers

How to Report File Metadata for 2000+ Files Uploaded by Multiple Users to Azure Blob Storage(Filename | Size | Uploaded By | Upload Time | Location)

Hi Team, I would know following. Multiple users are uploading multiple different files in blob storage, i need to extract metadata like who uploaded by,location,time,file type,file size. how to to get this. Filename | Size | Uploaded By | Upload Time |…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,534 questions
asked 2025-07-03T11:41:58.4033333+00:00
Prasad Sandu 0 Reputation points
commented 2025-07-04T13:39:39.02+00:00
Smaran Thoomu 24,575 Reputation points Microsoft External Staff Moderator
1 answer One of the answers was accepted by the question author.

How to create Databricks Cluster with the new Databricks Learner Account

Hello I have just signed up for the new Databricks Learner account to replace the traditional Databricks Community Edition. I am trying to create a traditional cluster, however it would appear I'm limited to just creating a SQL Warehouse, see image.…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,534 questions
asked 2025-07-03T10:02:10.8033333+00:00
Carlton Patterson 761 Reputation points
commented 2025-07-03T17:00:01.72+00:00
Venkat Reddy Navari 3,470 Reputation points Microsoft External Staff Moderator
1 answer One of the answers was accepted by the question author.

Databricks internal claude the model has been reporting errors today

the error is: {"error_code":"TEMPORARILY_UNAVAILABLE","message":"The server is currently unable to handle the request."}, is azure updating this service?

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,534 questions
asked 2025-06-28T03:13:12.6033333+00:00
davaid 60 Reputation points
commented 2025-07-03T16:38:34.0766667+00:00
Mayank Singh 0 Reputation points
1 answer One of the answers was accepted by the question author.

Differences Between Databricks Service Areas and Access Point Speed Limits

There are various service areas within Databricks, and clarification is needed on the differences between these areas, especially regarding usage with the Claude Model. Additionally, information is sought on whether Databricks imposes a speed limit at…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,534 questions
asked 2025-07-03T15:18:04.8+00:00
davaid 60 Reputation points
accepted 2025-07-03T16:20:35.33+00:00
davaid 60 Reputation points
4 answers

How to configure ADF pipeline run, linked service, so it uses Databricks serverless compute

Databricks has recently announced serverless compute for workflows: https://learn.microsoft.com/en-us/azure/databricks/workflows/jobs/run-serverless-jobs I would like to be able to execute Azure Data Factory (ADF) jobs using this…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,534 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,651 questions
asked 2024-05-01T12:12:06.9033333+00:00
Krzysztof Przysowa 20 Reputation points
answered 2025-07-03T09:06:41.83+00:00
Krzysztof Przysowa 20 Reputation points
1 answer One of the answers was accepted by the question author.

Query on Using Azure Free Account for Delta Live Tables (DLT) POC

We are currently exploring the latest features of Databricks Delta Live Tables (DLT) and are planning to build a small proof of concept (POC) to evaluate these capabilities. We request your guidance on the following: Free Azure Account Usage: Can we…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,534 questions
asked 2025-06-21T09:06:33.3133333+00:00
Janice Chi 160 Reputation points
accepted 2025-07-03T08:30:18.4366667+00:00
Janice Chi 160 Reputation points