Azure Databricks

0 answers

Not able to create the CREATE STORAGE CREDENTIAL as storage credential is not recognized command in azure databricks

I am trying to create the external location using the command . when I run the command then it does not recognize the storage credential then i can't run the external location command also. please help so that i can run the migrate table command in…

asked

padam 0

commented

padam 0

0 answers

Databricks internal claude the model has been reporting errors today

the error is: {"error_code":"TEMPORARILY_UNAVAILABLE","message":"The server is currently unable to handle the request."}, is azure updating this service?

asked

davaid 20

edited the question

davaid 20

1 answer

does Claude 4 in databricks supports reasoning?

i found claude 4 sonnet in azure databricks, but don't know how to enable thinking for this model

asked

davaid 20

accepted

davaid 20

1 answer

Not able to login to account level at azure databricks

I am working on Unity Catalog migration from hive metastore using UCX migration and I am not able to login to account level i.e. https://accounts.azuredatabricks.net. it redirects to workspace url after login . I need to go to account level login so that…

asked

padam 0

commented

Chandra Boorla 14,510 Microsoft External Staff Moderator

0 answers

Databricks Issue to use Lakebase / OLTP

Hello, We try to use new feature of Databricks regarding OLTP. we're having a problem. By following this documentation and doing the setup from the UI. https://learn.microsoft.com/en-us/azure/databricks/oltp/create/ …

asked

Maxime DEBOFFLE 0

commented

Chandra Boorla 14,510 Microsoft External Staff Moderator

0 answers

Guidance on Real-Time Streaming with Reconciliation and Domain Cutover Handling – Kafka → Databricks → Azure SQL Hyperscale

We are currently designing a real-time ingestion and reconciliation architecture using Databricks Structured Streaming that reads IBM DB2 CDC messages via Kafka and writes them to Azure SQL Hyperscale. The context is a healthcare data migration…

asked

Janice Chi 140

commented

Venkat Reddy Navari 2,885 Microsoft External Staff Moderator

0 answers

Can We View Last Month's DBU Usage by Job or Cluster in Azure Portal or Databricks UI

Can we view the total DBU (Databricks Unit) usage for the last month, broken down by cluster or job, directly in the Azure Portal or within the Azure Databricks UI (such as under Compute or any built-in usage section)? If not, what is the recommended…

asked

Sudhakar P 145

commented

Chandra Boorla 14,510 Microsoft External Staff Moderator

1 answer

I want to create an Azure Databricks in which I can connect to ADLS gen 2. Can anyone please help me as I am unable to do so or if anyone can provide steps end to end.

asked

Sanyog Vaidya 0

commented

Chandra Boorla 14,510 Microsoft External Staff Moderator

1 answer

Request for Merge Strategy Guidance in Databricks for CDC Payload from Kafka (IBM InfoSphere CDC)

We are currently designing a large-scale Change Data Capture (CDC) ingestion pipeline in a healthcare data platform, and we would appreciate your guidance regarding merge strategy performance and payload optimizations in Azure Databricks. 📌 Project…

asked

Janice Chi 140

commented

Chandra Boorla 14,510 Microsoft External Staff Moderator

1 answer

Unable to attach databricks workspace to unity catalog

Unable to attach data bricks workspace to unity catalog using my personel azure account

asked

Rishi Anand 0

commented

Venkat Reddy Navari 2,885 Microsoft External Staff Moderator

1 answer

Recommended Merge Strategy in Databricks for CDC Events from Kafka

We are implementing a Change Data Capture (CDC) pipeline where Kafka receives CDC events from IBM InfoSphere CDC. Each event is published as two consecutive flat JSON records: { "EMP_ID": 101 } — Primary Key { "SALARY": 9500 } —…

asked

Janice Chi 140

commented

Chandra Boorla 14,510 Microsoft External Staff Moderator

1 answer

Cursor position issue in Azure Databricks

Hi, when editing code inside Databricks notebook environment, the cursor does not position itself at where I thought I was typing. There seems to be difference in 5-6 character positions between where the cursor is and where characters are…

asked

Hong Ling TAN (NP) 40

accepted

Hong Ling TAN (NP) 40

1 answer

Workspace Admin gets 'Access Denied' when creating a cluster

Hello Azure Support Team, I am experiencing a critical "Access Denied" issue that is preventing me from creating any new compute clusters in my workspace. I am trying to create a new cluster from the Compute page, but I receive the error:…

asked

TrubrixAi Azure Admin 20

commented

Krupal Bandari 770 Microsoft External Staff Moderator

1 answer

best-practice framework for managing both Catch-up and Streaming pipelines (including reconciliation and retries) in Databricks

Details: In our project, once the one-time historical load and recon is completed, we start Catch-up ETL pipeline (processing CDC data from Kafka to Delta to Azure SQL) will be srated and Immediately after Catch-up ends, in parallel Catch-up…

asked

Janice Chi 140

commented

Chandra Boorla 14,510 Microsoft External Staff Moderator

2 answers

401 Unauthorized When Remounting ADLS/Blob Storage in Azure Databricks with Valid Service Principal

Hello, I recently generated a new client secret for our App Registration, as the previous one had expired. I then updated our Databricks Secret Scope using the following command: databricks secrets put --scope <secret-scope> --key…

asked

Ndjamen, P0462JN (601-ITC-Herden GmbH) 0

answered

Ndjamen, P0462JN (601-ITC-Herden GmbH) 0

1 answer

Best Practice for Reconciliation Row Count + Hash: Calculating in Databricks vs Pulling Precomputed Values from DB2 and Hyperscale

We are currently executing a one-time historical SQL data migration from an on-premises IBM DB2 (Snapshot Export) to Azure SQL Hyperscale via Azure Data Factory (ADF) for a healthcare analytics workload. The total data volume is approximately 80 TB,…

asked

Janice Chi 140

commented

Venkat Reddy Navari 2,885 Microsoft External Staff Moderator

1 answer

Azure Databricks vs. DLT for Near Real-Time Streaming

We are currently designing a Near Real-Time (NRT) streaming pipeline for a healthcare analytics workload and would appreciate your guidance to validate the architecture and compute planning. Project Context: Ingestion Source: Kafka-based CDC…

asked

Janice Chi 140

commented

Venkat Reddy Navari 2,885 Microsoft External Staff Moderator

1 answer

Cost, Scaling, and Migration Considerations – Spark Structured Streaming (DBR) vs. Delta Live Tables (DLT)

We are currently designing a near real-time streaming pipeline for a healthcare analytics workload and are evaluating Databricks Spark Structured Streaming (using DBR) versus Delta Live Tables (DLT) for implementation. Project Context: Ingestion source:…

asked

Janice Chi 140

answered

Venkat Reddy Navari 2,885 Microsoft External Staff Moderator

1 answer

Clarification on Autoscaling Limitations for Structured Streaming and Use of DLT in Azure Databricks

In our current streaming architecture built on Azure Databricks, we are using Structured Streaming workloads running on auto-scaling clusters. However, we’ve noticed in below link…

asked

Janice Chi 140

commented

Smaran Thoomu 24,110 Microsoft External Staff Moderator

2 answers

created azure databricks workspace with Global Admin (Tenant level) but workspace is not created with Workspace admin role. how can i be workspace admin.?

I created azure databricks workspace from my personal account (NOT an organization account) . i am global Admin at the tenant level. Workspace is not showing me as a Workspace admin My Name is incorrect in the workspace at the user icon at the top…

asked

srinivasu machineni 0

commented

Smaran Thoomu 24,110 Microsoft External Staff Moderator

Azure Databricks

Filter

Content

2,514 questions with Azure Databricks tags

Not able to create the CREATE STORAGE CREDENTIAL as storage credential is not recognized command in azure databricks

Databricks internal claude the model has been reporting errors today

does Claude 4 in databricks supports reasoning?

Not able to login to account level at azure databricks

Databricks Issue to use Lakebase / OLTP

Guidance on Real-Time Streaming with Reconciliation and Domain Cutover Handling – Kafka → Databricks → Azure SQL Hyperscale

Can We View Last Month's DBU Usage by Job or Cluster in Azure Portal or Databricks UI

Azure Databricks

Request for Merge Strategy Guidance in Databricks for CDC Payload from Kafka (IBM InfoSphere CDC)

Unable to attach databricks workspace to unity catalog

Recommended Merge Strategy in Databricks for CDC Events from Kafka

Cursor position issue in Azure Databricks

Workspace Admin gets 'Access Denied' when creating a cluster

best-practice framework for managing both Catch-up and Streaming pipelines (including reconciliation and retries) in Databricks

401 Unauthorized When Remounting ADLS/Blob Storage in Azure Databricks with Valid Service Principal

Best Practice for Reconciliation Row Count + Hash: Calculating in Databricks vs Pulling Precomputed Values from DB2 and Hyperscale

Azure Databricks vs. DLT for Near Real-Time Streaming

Cost, Scaling, and Migration Considerations – Spark Structured Streaming (DBR) vs. Delta Live Tables (DLT)

Clarification on Autoscaling Limitations for Structured Streaming and Use of DLT in Azure Databricks

created azure databricks workspace with Global Admin (Tenant level) but workspace is not created with Workspace admin role. how can i be workspace admin.?