Azure Databricks

0 answers

Orchastration Control in Data Pipeline in Azure

I am designing the Data Pipeline which Consume Data from Salesforce using Bulk API endpoint (pull mechanism). The Data comes and land to ADLS Gen2 Bronze Layer. Next Transformation job will start and clean the data and push to Silver layer ADLS…

asked

Relay 200

edited the question

Relay 200

0 answers

Unable to Create Compute Cluster in Azure Databricks

Currently learning Data Engineering with Azure and using the trial version with $200 credit. While attempting to create a cluster in Azure Databricks, errors are encountered. The following steps have been taken: Changed locations (East US and West US…

asked

Vyshnavi Moparthi 0

1 answer

how do i connect with Databricks from Azure Analysis Service

We have our processed/gold layer data in databricks catalogue. We need to publish this data in to Azure Analysis Services. How I connect to a databricks table and pull data directly into AAS for our daily data refresh (processing). The data is being…

asked

salmanahmed-2826 0

answered

Alex Burlachenko 11,610

1 answer

Error while using DBUtils command in Databricks notebook to show list of file from data lake storage account

Hi All, I am getting error while using DBUTILS command from databricks to show list of files from data lake storage account. Below is error screen shot. It is showing error invalid character in host name. What i tried so far. Checked all permission on…

asked

Davendra Kumar 0

commented

Smaran Thoomu 25,115 Microsoft External Staff Moderator

1 answer

Databricks Execution Strategy for One-Time 80 TB Migration

In our one-time historical data migration project, we are processing approximately 80 TB of CSV data into Parquet format using Azure Databricks, within a strict 3-day completion window. The raw CSV files are extracted from on-prem DB2 snapshots and…

asked

Janice Chi 320

edited a comment

Smaran Thoomu 25,115 Microsoft External Staff Moderator

1 answer

Error while accessing Azure portal

Hi, I'm getting the below error after I log to the Azure portal Selected user account does not exist in tenant 'Microsoft Services' and cannot access the application 'e6694c91-1590-4e35-9bb7-b865c638b9c1' in that tenant. The account needs to be added as…

asked

Anu Merrin Bijoy 0

answered

Deepanshu katara 17,025 MVP Moderator

1 answer

Cluster Creation Flow in ADF-Databricks Integration Using Instance Pools

We are executing a one-time historical data migration (~80 TB) from on-prem DB2 to Azure SQL Hyperscale, using ADF to orchestrate Databricks for CSV-to-Parquet conversion. Based on partition volume and estimated table size, we have three ADF pipelines…

asked

Janice Chi 320

answered

Venkat Reddy Navari 3,630 Microsoft External Staff Moderator

1 answer

Getting error while launching a cluster for Azure Databricks with the free trial.

Hello, I I am using a student account for Azure. When I try to launch the cluster of Azure Databricks, it is failing by saying the error code as: The VM size you are specifying is not available. [details] SKU NotAvailable: The requested VM size for…

asked

Ambareesh Reddy Polu 0

commented

Venkat Reddy Navari 3,630 Microsoft External Staff Moderator

1 answer

Databricks Workspace is not lading.

Databricks Workspace is not loading/ too slow. When login into workspace, it's not loading.

asked

Adhish Manu Valsalakumar 0

commented

Venkat Reddy Navari 3,630 Microsoft External Staff Moderator

1 answer

Tag is not running the repo and instead only the workspace

Jobs are running for only the repo and not including the workspace. What is causing this to occur in Databricks?

asked

Kurt 0

commented

Kurt 0

0 answers

Can Databricks (Customer-Provided Workspace) Access ADLS Gen2 Using Managed Identity?

We are working on a healthcare-focused data migration project involving IBM DB2 to Azure SQL Hyperscale, with intermediate storage in ADLS Gen2 (Parquet format). The migration involves: One-time historical load via FlashCopy Catch-up and real-time…

asked

Janice Chi 320

commented

Venkat Reddy Navari 3,630 Microsoft External Staff Moderator

1 answer

Efficient Data Migration of Databricks from One Region to Another

How can data bricks be migrated from US East 2 to US Central efficiently? The migration includes moving the existing dev and prod setup, which consists of workspace jobs, notebooks, scripts, catalog tables, volumes, and Azure Data Factory (ADF)…

asked

Deepa M 20

accepted

Deepa M 20

1 answer

Unable to start Databricks cluster using Azure free credit account

I have been trying to start a cluster in Databricks using Azure. Since I am using a azure free account I know that I can only use 4 cores and a single node in Databricks, but even with this configuration I get the following error message: Cloud…

asked

Arjun Thonoor 0

commented

Smaran Thoomu 25,115 Microsoft External Staff Moderator

1 answer

Convert my personal account's Eligible Owner role to Active Owner on my Pay-as-you-go subscription

I’m currently learning Azure Databricks and encountered a critical permissions issue. I created a Pay-as-you-go Azure subscription under my personal Microsoft account. I then created a work email and separate Entra ID tenant to manage Databricks and…

asked

yin li 0

commented

Smaran Thoomu 25,115 Microsoft External Staff Moderator

1 answer

Best Practices and Risks for Row Number–Based Partitioning in One-Time DB2 Snapshot Load

We are performing a large-scale one-time historical migration (~80 TB) from on-prem IBM DB2 (via FlashCopy snapshot) into Azure SQL Hyperscale, using the following flow: Extract via SHIR into ADLS Gen2 (CSV format), Transform into Parquet and add…

asked

Janice Chi 320

commented

Smaran Thoomu 25,115 Microsoft External Staff Moderator

2 answers

How to set up databricks-bundle in Azure Databricks UI

Hello Mr./Ms., I'm a new bee with Databircks and databricks-bundle for CICD. I have 3 questions:. By chance, I saw this picture: I tried to look for documents or videos related to Databricks Bundles in the UI (like in the image above), including how…

asked

Viet Tran 70

commented

Viet Tran 70

1 answer

This message was deleted a while ago. This message was deleted a while ago

asked

HimashaPinto-3528 0

edited a comment

HimashaPinto-3528 0

1 answer

Unable to create cluster. "Compute Terminated" is the error message.

Unable to create an all-purpose compute cluster. I made multiple attempts with different accounts, cluster configurations, and regions. But I am unable to create a cluster.

asked

Kapil Chogga 20

commented

Kapil Chogga 20

1 answer