Microsoft Q&A

Azure Databricks

1,288 questions

An Apache Spark-based analytics platform optimized for Azure.

Browse all Azure tags

1,288 questions with Azure Databricks tags

Sort by: Updated
1 answer

How to call notbook cmds in another notebook

Hello Team, The notebook called "abc" and in that notebook there are 2 cmd for example mentioned below I need to call this cmd in another notebook cmd. how it can be done. Please advise Regards Rohit

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,288 questions
asked 2023-03-21T11:17:07.93+00:00
Rohit Kulkarni 656 Reputation points
commented 2023-03-24T20:51:11.1633333+00:00
MartinJaffer-MSFT 25,556 Reputation points Microsoft Employee
1 answer

Grant only one user access to storage account from a shared Databricks workspace

I have a user on a Databricks workspace that I want to allow to read data from a storage account: sto-limited-access. I do not want any of the other users on that workspace to be able to read the data. From all I have read I have not found any good…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,288 questions
Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
1,527 questions
asked 2023-03-23T09:41:23.1433333+00:00
Mattias Jorstedt 20 Reputation points
edited an answer 2023-03-24T09:50:16.8633333+00:00
Mattias Jorstedt 20 Reputation points
0 answers

Issue reading delta in storage account gen-2

I was reading a delta table in storage account gen 2 on azure Databricks with spark and today the command didn't work and I'm receiving timeout error, it just continue running the command: spark.read.format('delta').load(my table)

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,288 questions
Azure Storage Accounts
Azure Storage Accounts
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
1,673 questions
asked 2023-03-22T19:37:19.9766667+00:00
Ferreira, Jeniffer (NAZ-V) 0 Reputation points
commented 2023-03-23T23:44:31.35+00:00
BhargavaGunnam-MSFT 10,326 Reputation points Microsoft Employee
1 answer

Medallion architecture, how to update Gold tables?

Assume that I have a data source that is ingested to a few bronze tables, and transformed to a silver table. Ans next, a gold table is created by aggregating the silver table. If new records arrive in the data source, bronze and silver tables are updated…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,288 questions
asked 2023-03-07T12:30:20.6433333+00:00
Mohammad Saber 406 Reputation points
edited a comment 2023-03-23T21:31:57.4+00:00
Mohammad Saber 406 Reputation points
1 answer

how to delete a folder in ADLS with different name

Hi, I have a pipeline who generate parquet file in ADLS gen2 with similar naming patternfor each folder every day. bust some times a bug cause an empty folder generate in all folders with different naming. I would like to use delete activity befor my…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,288 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
2,850 questions
Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
930 questions
asked 2023-03-13T14:15:03.5433333+00:00
17285674 0 Reputation points
edited the question 2023-03-23T20:18:25.6766667+00:00
MuthuKumaranMurugaachari-MSFT 8,566 Reputation points
1 answer One of the answers was accepted by the question author.

Use explicitly Unity catalog instead of hive metastore

I have a customer who is using Databricks with Unity Catalog metastore. All outbound internet traffic is blocked through firewall. When trying to run a job, databricks is trying to make a connection to…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,288 questions
asked 2023-03-22T09:34:22.52+00:00
George Alexiou 20 Reputation points Microsoft Employee
accepted 2023-03-23T08:38:31.15+00:00
George Alexiou 20 Reputation points Microsoft Employee
1 answer

Do we have direct connector to connect delta share(Delta table) and load data into Azure SQL database?

Description: Using Delta Share Concept, we tried to load the data from Data brick notebook by creating Delta Share CatLog. Data provider has given the delta table access to Data receiver. However this approach need to built Data brick Notebook and need…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,288 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,670 questions
Azure SQL Database
Azure Data Share
Azure Data Share
An Azure service that is used to share data from multiple sources with other organizations.
37 questions
asked 2023-03-20T11:18:44.07+00:00
Ravi Alwal 0 Reputation points
commented 2023-03-23T07:34:26.91+00:00
PRADEEPCHEEKATLA-MSFT 56,301 Reputation points Microsoft Employee
0 answers

Azure Databricks ->Data Explorer -> Hive Table -> Cannot See Create Query Option

I have a pay as you go subscription. Have created a schema and created table by importing csv When I navigate as per below Azure Databricks ->Data Explorer -> Hive Table -> Cannot See Create Query Option I click on the table, on top right side…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,288 questions
asked 2023-03-20T06:08:23.84+00:00
Ashish P 0 Reputation points
commented 2023-03-23T07:32:41.5966667+00:00
PRADEEPCHEEKATLA-MSFT 56,301 Reputation points Microsoft Employee
1 answer

Azure Databricks - How to calculate the cost for a job that is configured to use a job cluster?

Azure Databricks - How do i calculate the cost of a databricks job that uses a job cluster? How do I get the number of workers being utilized for a job in real time? Is the cost dependant on the number of workers that are up and running?

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,288 questions
Azure Cost Management
Azure Cost Management
A Microsoft offering that enables tracking of cloud usage and expenditures for Azure and other cloud providers.
1,081 questions
asked 2023-03-09T14:14:11.49+00:00
Anusha S Raj 21 Reputation points
commented 2023-03-23T07:22:28.24+00:00
PRADEEPCHEEKATLA-MSFT 56,301 Reputation points Microsoft Employee
1 answer One of the answers was accepted by the question author.

Can't Create a New Unity Catalog in Databricks

I am unable to create a new metastore which prevents me from creating a unity catalog from the Databricks UI. As the owner of the workspace, I have logged in and then clicked the data area and I see no affordance like a button or link that allows me to…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,288 questions
asked 2023-01-06T17:20:25.437+00:00
Brian Custer 21 Reputation points
commented 2023-03-23T07:10:19.47+00:00
PRADEEPCHEEKATLA-MSFT 56,301 Reputation points Microsoft Employee
1 answer

Databricks with private link

hi, i 'm following this guide to set up secured a databricks cluster: https://learn.microsoft.com/en-us/azure/databricks/administration-guide/cloud-configurations/azure/private-link#dns So i only have 1 vnet and 3 subnet inside: public-subnet,…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,288 questions
asked 2022-10-14T04:27:24.083+00:00
Phuong Dinh 1 Reputation point
edited the question 2023-03-23T07:07:42.63+00:00
PRADEEPCHEEKATLA-MSFT 56,301 Reputation points Microsoft Employee
1 answer One of the answers was accepted by the question author.

Metastore table creation issue

I am following the process mentioned in the below document but unable to create table in the metastore https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/get-started  what could be the root cause for this error. Error in…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,288 questions
asked 2023-03-20T01:32:03.2233333+00:00
Kumari s 40 Reputation points
accepted 2023-03-22T20:09:28.7966667+00:00
Kumari s 40 Reputation points
1 answer One of the answers was accepted by the question author.

Access Control using Service Principal in the databricks

In order for our Databricks workspace to access different data sets, we need a way to grant access control either at an individual or role level. These data sets are saved as files in Data Lake Gen2 and can be organized to align with our access rights…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,288 questions
asked 2023-03-20T01:38:12.6466667+00:00
Kumari s 40 Reputation points
accepted 2023-03-22T20:09:16.65+00:00
Kumari s 40 Reputation points
2 answers

Azure Data Factory User Configuration Issues for Empty URI

I am developing the ADF pipelines with calling azure databricks notebook. in that notebook, I am passing the API url to notebook to execute our pyspark code. it is working normal debugging mode. if we try to moved DEV/Main branch with trigger now or…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,288 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,670 questions
asked 2023-03-17T10:24:07.7933333+00:00
Karthik Elavan 1 Reputation point
commented 2023-03-22T17:40:55.0433333+00:00
MartinJaffer-MSFT 25,556 Reputation points Microsoft Employee
1 answer

Migration of full resource group from Dev to UAT within same subscription

I am migrating data from azure DEV environment to azure UAT environment. below are the components which i need to migrate;- Azure blob storage- Dev envt. Azure Logic Apps - Dev Envt Azure SQL database- dev envt. DataBricks - Dev Env I went through…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,288 questions
Azure Logic Apps
Azure Logic Apps
An Azure service that automates the access and use of data across clouds without writing code.
1,960 questions
asked 2023-03-21T11:10:43.0466667+00:00
Sabareetha M P 0 Reputation points
edited the question 2023-03-22T12:47:22.52+00:00
SadiqhAhmed-MSFT 25,781 Reputation points Microsoft Employee
0 answers

Unable to mount ADLS

Hi All, I am trying to mount ADLS to Azure Databricks but getting this error - "shaded.databricks.org.apache.hadoop.fs.azure.AzureException: shaded.databricks.org.apache.hadoop.fs.azure.AzureException: Unable to access container scd in account…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,288 questions
Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
930 questions
Azure Active Directory
Azure Active Directory
An Azure enterprise identity service that provides single sign-on and multi-factor authentication.
13,549 questions
asked 2023-03-17T16:22:46.6333333+00:00
Sarvesh Pandey 71 Reputation points
commented 2023-03-21T22:03:29.21+00:00
BhargavaGunnam-MSFT 10,326 Reputation points Microsoft Employee
1 answer One of the answers was accepted by the question author.

Overwriting the existing table in Databricks; Mechanism and History?

Hi, Assume that I have a delta table stored on an Azure storage account. When new records arrive, I repeat the transformation and overwrite the existing table. (DF.write .format("delta") .mode("overwrite") …

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,288 questions
asked 2023-03-09T03:48:00.64+00:00
Mohammad Saber 406 Reputation points
accepted 2023-03-21T21:29:48.2133333+00:00
Mohammad Saber 406 Reputation points
1 answer

Databricks Resource Limit

We are currently blocked from creating a databricks cluster, we are facing a resource limit issue for vCPU and other resources and we could not resolve this with Azure support.

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,288 questions
asked 2023-03-20T18:49:08.3633333+00:00
Chuk Amattah 0 Reputation points
answered 2023-03-21T19:45:48.9733333+00:00
HimanshuSinha-msft 18,366 Reputation points Microsoft Employee
3 answers One of the answers was accepted by the question author.

Copy Delta File from one ADLS to another ADLS

Hi Team, I have Delta file in one ADLS and need to copy to another, can I copy it using ADF ?

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,288 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,670 questions
asked 2023-03-16T11:51:35.0833333+00:00
Sharukh Kundagol 105 Reputation points
commented 2023-03-21T10:31:04.1966667+00:00
PRADEEPCHEEKATLA-MSFT 56,301 Reputation points Microsoft Employee
1 answer One of the answers was accepted by the question author.

Integrating Application insight with Azure Databricks and Azure DataFactory

Can we Integrate Application insight with Azure Data bricks and Azure Data Factory?

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,288 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,670 questions
asked 2023-02-23T19:30:54.8666667+00:00
Sharukh Kundagol 105 Reputation points
commented 2023-03-21T10:30:56.9066667+00:00
PRADEEPCHEEKATLA-MSFT 56,301 Reputation points Microsoft Employee