Azure Databricks

1 answer

Azure Data Bricks Cluster

Hi, While creating Azure Databricks cluster I am getting an error that the node Standrad_DS3_vs is not available in the Region. Please help. I am not able to see any Region while I run a Azure bash command to find the Region which has Standard_DS3_v2…

asked

Rishi Rithi 0

accepted

Rishi Rithi 0

2 answers

How to implement retry logic in Azure Databricks for a failed microbatch in Structured Streaming (without replaying from Kafka)?

We are ingesting CDC data from Kafka into Azure Databricks using Structured Streaming. In rare cases, a microbatch fails due to reconciliation logic (e.g., row count or hash mismatch). We would like to reprocess only the failed batch (e.g., write the…

asked

Gabriel25 545

accepted

Gabriel25 545

1 answer

how to connect onprem hadoop hive to azure databricks

asked

Prasad Sandu 0

commented

Smaran Thoomu 24,575 Microsoft External Staff Moderator

1 answer

Unable to access account console

We are currently unable to access the Databricks Account Console (https://accounts.azuredatabricks.net) for our Azure Databricks deployment. Attempts to log in using our Azure AD Global Administrator account redirect us to the workspace instead of the…

asked

Chaithanya Chowdhary 0

commented

Smaran Thoomu 24,575 Microsoft External Staff Moderator

0 answers

Not able to open the databricks work space(adb-1457270792635633.13.azuredatabricks.net)

I am not able to open the azure data bricks work space (adb-1457270792635633.13.azuredatabricks.net). Need your help on it pls .

asked

Yakkli, Bala Ramesh 0

commented

Smaran Thoomu 24,575 Microsoft External Staff Moderator

1 answer

How to set up databricks-bundle in Azure Databricks UI

Hello Mr./Ms., I'm a new bee with Databircks and databricks-bundle for CICD. I have 3 questions:. By chance, I saw this picture: I tried to look for documents or videos related to Databricks Bundles in the UI (like in the image above), including how…

asked

Viet Tran 50

commented

Viet Tran 50

2 answers

Best practices for secure and performant JDBC access from Azure Databricks to on-prem Hadoop Hive over VPN

We have a VNet-injected Azure Databricks workspace connected to on-prem Hadoop via VPN Gateway. We want to read from Hive tables using the JDBC driver in Databricks notebooks. While the connection works, performance is slow and we're unsure about the…

asked

Vikranth-2626 160

accepted

Vikranth-2626 160

1 answer

How to selectively re-run only failed ETL phases (extract/transform/load) for specific partitions in a continuous pipeline?

Question: In our current reconciliation framework, we log the success/failure of each ETL pipeline phase—extraction, transformation, and load—using dedicated status flags (extraction_status, transformation_status, load_status) in a phase-level status…

asked

Janice Chi 160

commented

Smaran Thoomu 24,575 Microsoft External Staff Moderator

1 answer

How to disable serverless culster in Azure Databricks?

How to disable serverless culster in Azure Databricks? Mail to Suport team asking paid subscription

asked

Akash Nidavani 0

commented

Smaran Thoomu 24,575 Microsoft External Staff Moderator

1 answer

Best Retry Strategy for Reconciliation-Failed Microbatches in Real-Time CDC Pipelines (Azure SQL + Databricks)

We are implementing a near-real-time CDC streaming pipeline using Azure Databricks and Azure SQL Hyperscale. Flattened CDC data from Kafka is ingested into Delta Lake (ADLS) per microbatch, and each batch is also written to a staging table in Azure SQL.…

asked

Janice Chi 160

commented

Smaran Thoomu 24,575 Microsoft External Staff Moderator

1 answer

Databricks Execution Strategy for One-Time 80 TB Migration

In our one-time historical data migration project, we are processing approximately 80 TB of CSV data into Parquet format using Azure Databricks, within a strict 3-day completion window. The raw CSV files are extracted from on-prem DB2 snapshots and…

asked

Janice Chi 160

commented

Smaran Thoomu 24,575 Microsoft External Staff Moderator

0 answers

The workspace is in a failed state and cannot be launched.

When I try to delete a Databricks resource - I get the error... The workspace 'workspace_name' is in a failed state and cannot be launched. Please review error details in the activity log tab and retry your operation. How do I resolve this? I've…

asked

Sukhminder Bining 0

commented

Sukhminder Bining 0

0 answers

Azure Databricks pytorch error

Though i have check Machine learning checkbox. When i try to run BartForConditionalGeneration it shows Pytorch not installed. I saw the support page they have mentioned to install pytorch for CPU based clusters with wheel but where and how should i run…

asked

Aryan Rastogi 60

commented

Venkat Reddy Navari 3,470 Microsoft External Staff Moderator

1 answer

Using Dynamic Job Parameters with Databricks Job Activity

A replacement is being implemented for a generic Databricks job trigger pipeline with the new Databricks job activity. The goal is to maintain all job triggering logic in a separate pipeline. Currently, a JSON-like string is being dynamically passed with…

asked

Víctor C. Fernández 0

commented

Venkat Reddy Navari 3,470 Microsoft External Staff Moderator

2 answers

How to Report File Metadata for 2000+ Files Uploaded by Multiple Users to Azure Blob Storage(Filename | Size | Uploaded By | Upload Time | Location)

Hi Team, I would know following. Multiple users are uploading multiple different files in blob storage, i need to extract metadata like who uploaded by,location,time,file type,file size. how to to get this. Filename | Size | Uploaded By | Upload Time |…

asked

Prasad Sandu 0

commented

Smaran Thoomu 24,575 Microsoft External Staff Moderator

1 answer

How to create Databricks Cluster with the new Databricks Learner Account

Hello I have just signed up for the new Databricks Learner account to replace the traditional Databricks Community Edition. I am trying to create a traditional cluster, however it would appear I'm limited to just creating a SQL Warehouse, see image.…

asked

Carlton Patterson 761

commented

Venkat Reddy Navari 3,470 Microsoft External Staff Moderator

1 answer

Databricks internal claude the model has been reporting errors today

the error is: {"error_code":"TEMPORARILY_UNAVAILABLE","message":"The server is currently unable to handle the request."}, is azure updating this service?

asked

davaid 60

commented

Mayank Singh 0

1 answer

Differences Between Databricks Service Areas and Access Point Speed Limits

There are various service areas within Databricks, and clarification is needed on the differences between these areas, especially regarding usage with the Claude Model. Additionally, information is sought on whether Databricks imposes a speed limit at…

asked

davaid 60

accepted

davaid 60

4 answers

How to configure ADF pipeline run, linked service, so it uses Databricks serverless compute

Databricks has recently announced serverless compute for workflows: https://learn.microsoft.com/en-us/azure/databricks/workflows/jobs/run-serverless-jobs I would like to be able to execute Azure Data Factory (ADF) jobs using this…

asked

Krzysztof Przysowa 20

answered

Krzysztof Przysowa 20

1 answer

Query on Using Azure Free Account for Delta Live Tables (DLT) POC

We are currently exploring the latest features of Databricks Delta Live Tables (DLT) and are planning to build a small proof of concept (POC) to evaluate these capabilities. We request your guidance on the following: Free Azure Account Usage: Can we…

asked

Janice Chi 160

accepted

Janice Chi 160

Filter

Content

2,534 questions with Azure Databricks tags

Azure Data Bricks Cluster

How to implement retry logic in Azure Databricks for a failed microbatch in Structured Streaming (without replaying from Kafka)?

how to connect onprem hadoop hive to azure databricks

Unable to access account console

Not able to open the databricks work space(adb-1457270792635633.13.azuredatabricks.net)

How to set up databricks-bundle in Azure Databricks UI

Best practices for secure and performant JDBC access from Azure Databricks to on-prem Hadoop Hive over VPN

How to selectively re-run only failed ETL phases (extract/transform/load) for specific partitions in a continuous pipeline?

How to disable serverless culster in Azure Databricks?

Best Retry Strategy for Reconciliation-Failed Microbatches in Real-Time CDC Pipelines (Azure SQL + Databricks)

Databricks Execution Strategy for One-Time 80 TB Migration

The workspace is in a failed state and cannot be launched.

Azure Databricks pytorch error

Using Dynamic Job Parameters with Databricks Job Activity

How to Report File Metadata for 2000+ Files Uploaded by Multiple Users to Azure Blob Storage(Filename | Size | Uploaded By | Upload Time | Location)

How to create Databricks Cluster with the new Databricks Learner Account

Databricks internal claude the model has been reporting errors today

Differences Between Databricks Service Areas and Access Point Speed Limits

How to configure ADF pipeline run, linked service, so it uses Databricks serverless compute

Query on Using Azure Free Account for Delta Live Tables (DLT) POC