Azure Data Bricks Cluster
Hi, While creating Azure Databricks cluster I am getting an error that the node Standrad_DS3_vs is not available in the Region. Please help. I am not able to see any Region while I run a Azure bash command to find the Region which has Standard_DS3_v2…
Azure Databricks
How to implement retry logic in Azure Databricks for a failed microbatch in Structured Streaming (without replaying from Kafka)?
We are ingesting CDC data from Kafka into Azure Databricks using Structured Streaming. In rare cases, a microbatch fails due to reconciliation logic (e.g., row count or hash mismatch). We would like to reprocess only the failed batch (e.g., write the…
Azure Databricks
how to connect onprem hadoop hive to azure databricks
how to connect onprem hadoop hive to azure databricks
Azure Databricks
Unable to access account console
We are currently unable to access the Databricks Account Console (https://accounts.azuredatabricks.net) for our Azure Databricks deployment. Attempts to log in using our Azure AD Global Administrator account redirect us to the workspace instead of the…
Azure Databricks
Not able to open the databricks work space(adb-1457270792635633.13.azuredatabricks.net)
I am not able to open the azure data bricks work space (adb-1457270792635633.13.azuredatabricks.net). Need your help on it pls .
Azure Databricks
How to set up databricks-bundle in Azure Databricks UI
Hello Mr./Ms., I'm a new bee with Databircks and databricks-bundle for CICD. I have 3 questions:. By chance, I saw this picture: I tried to look for documents or videos related to Databricks Bundles in the UI (like in the image above), including how…
Azure Databricks
Best practices for secure and performant JDBC access from Azure Databricks to on-prem Hadoop Hive over VPN
We have a VNet-injected Azure Databricks workspace connected to on-prem Hadoop via VPN Gateway. We want to read from Hive tables using the JDBC driver in Databricks notebooks. While the connection works, performance is slow and we're unsure about the…
Azure Databricks
How to selectively re-run only failed ETL phases (extract/transform/load) for specific partitions in a continuous pipeline?
Question: In our current reconciliation framework, we log the success/failure of each ETL pipeline phase—extraction, transformation, and load—using dedicated status flags (extraction_status, transformation_status, load_status) in a phase-level status…
Azure Databricks
How to disable serverless culster in Azure Databricks?
How to disable serverless culster in Azure Databricks? Mail to Suport team asking paid subscription
Azure Databricks
Best Retry Strategy for Reconciliation-Failed Microbatches in Real-Time CDC Pipelines (Azure SQL + Databricks)
We are implementing a near-real-time CDC streaming pipeline using Azure Databricks and Azure SQL Hyperscale. Flattened CDC data from Kafka is ingested into Delta Lake (ADLS) per microbatch, and each batch is also written to a staging table in Azure SQL.…
Azure Databricks
Databricks Execution Strategy for One-Time 80 TB Migration
In our one-time historical data migration project, we are processing approximately 80 TB of CSV data into Parquet format using Azure Databricks, within a strict 3-day completion window. The raw CSV files are extracted from on-prem DB2 snapshots and…
Azure Databricks
The workspace is in a failed state and cannot be launched.
When I try to delete a Databricks resource - I get the error... The workspace 'workspace_name' is in a failed state and cannot be launched. Please review error details in the activity log tab and retry your operation. How do I resolve this? I've…
Azure Databricks
Azure Databricks pytorch error
Though i have check Machine learning checkbox. When i try to run BartForConditionalGeneration it shows Pytorch not installed. I saw the support page they have mentioned to install pytorch for CPU based clusters with wheel but where and how should i run…
Azure Databricks
Using Dynamic Job Parameters with Databricks Job Activity
A replacement is being implemented for a generic Databricks job trigger pipeline with the new Databricks job activity. The goal is to maintain all job triggering logic in a separate pipeline. Currently, a JSON-like string is being dynamically passed with…
Azure Databricks

How to Report File Metadata for 2000+ Files Uploaded by Multiple Users to Azure Blob Storage(Filename | Size | Uploaded By | Upload Time | Location)
Hi Team, I would know following. Multiple users are uploading multiple different files in blob storage, i need to extract metadata like who uploaded by,location,time,file type,file size. how to to get this. Filename | Size | Uploaded By | Upload Time |…
Azure Databricks
How to create Databricks Cluster with the new Databricks Learner Account
Hello I have just signed up for the new Databricks Learner account to replace the traditional Databricks Community Edition. I am trying to create a traditional cluster, however it would appear I'm limited to just creating a SQL Warehouse, see image.…
Azure Databricks
Databricks internal claude the model has been reporting errors today
the error is: {"error_code":"TEMPORARILY_UNAVAILABLE","message":"The server is currently unable to handle the request."}, is azure updating this service?
Azure Databricks
Differences Between Databricks Service Areas and Access Point Speed Limits
There are various service areas within Databricks, and clarification is needed on the differences between these areas, especially regarding usage with the Claude Model. Additionally, information is sought on whether Databricks imposes a speed limit at…
Azure Databricks
How to configure ADF pipeline run, linked service, so it uses Databricks serverless compute
Databricks has recently announced serverless compute for workflows: https://learn.microsoft.com/en-us/azure/databricks/workflows/jobs/run-serverless-jobs I would like to be able to execute Azure Data Factory (ADF) jobs using this…
Azure Databricks
Azure Data Factory
Query on Using Azure Free Account for Delta Live Tables (DLT) POC
We are currently exploring the latest features of Databricks Delta Live Tables (DLT) and are planning to build a small proof of concept (POC) to evaluate these capabilities. We request your guidance on the following: Free Azure Account Usage: Can we…