2,211 questions with Azure Databricks tags

Sort by: Updated
1 answer

Azure Databricks is not sending email when Databricks Alert condition is met

Azure Databricks is not sending an email to the destination email address when Databricks Alert condition is met, Sometimes this is working and sometimes not.

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,211 questions
asked 2024-11-01T09:50:13.5066667+00:00
Mekala, Vamshi 0 Reputation points
answered 2024-11-02T14:00:00.4433333+00:00
Amira Bedhiafi 25,946 Reputation points
1 answer

Azure Databricks Workspace: Users are being added to the admins group on their own

We have an Azure Databricks workspace with Unity Catalog enabled. We are using SCIM provisioning to sync some Entra security groups to the workspace so that we can control access to the catalog items. We have noticed that some users are being added…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,211 questions
asked 2024-11-01T10:04:34.28+00:00
Chris O'Brien 0 Reputation points
answered 2024-11-02T13:58:14.5833333+00:00
Amira Bedhiafi 25,946 Reputation points
2 answers

Error when creating Access Connector for Azure Databricks

So, just created storage, and would like to setup the access connector, and faces the below error. Anyone seen this before ? All insights are very much appreciated

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,211 questions
asked 2024-10-31T10:38:45.89+00:00
Anders Thøgersen 0 Reputation points
commented 2024-11-02T13:49:55.7533333+00:00
Amira Bedhiafi 25,946 Reputation points
1 answer

I dont see Account Console in Databricks. Please advise

I have personal account and when I log into databricks I dont see account console to admin stuff

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,211 questions
asked 2024-10-31T13:46:23.6+00:00
Gokhan Erkus 0 Reputation points
commented 2024-11-02T11:16:14.41+00:00
Vinodh247 22,871 Reputation points
1 answer

Azure Databricks Compute Instance Quota Issue

Unable to create a compute instance in Azure Databricks due to a quota issue. The error message received is: This account may not have enough CPU cores to start a cluster. Contact your administrator to increase the limits. Additionally, it is noted…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,211 questions
asked 2024-10-30T17:40:56.9433333+00:00
Dmitriy Khodyrev 0 Reputation points
commented 2024-11-02T06:48:27.9433333+00:00
Naagesh R Aswartha 0 Reputation points
2 answers

Not able to create compute in Databricks on Azure Free Trialaccount

Hi Team, I am not able to create compute in Databricks on Azure Free Trial account. I tried all options for creating Databricks workspace - 14 days trial, standard account, premium account - but getting same error in all of these. Error message- This…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,211 questions
asked 2024-10-28T17:12:40.6533333+00:00
techie1234 5 Reputation points
commented 2024-11-01T20:28:07.6433333+00:00
Ganesh Gurram 945 Reputation points Microsoft Vendor
2 answers

How to fix the below issue which I am facing while installing the maven library with coordinates as 'com.databricks:spark-xml_2.12:0.14.0'

How to fix the below issue which I am facing while installing the maven library with coordinates as 'com.databricks:spark-xml_2.12:0.14.0' in ADF pipeline in azure databricks notebook activity run failed with error message Library installation…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,211 questions
asked 2024-10-27T08:34:14.07+00:00
Dathathraiah 0 Reputation points
commented 2024-11-01T09:17:55.4666667+00:00
Vinodh247 22,871 Reputation points
1 answer One of the answers was accepted by the question author.

Automating conversion of python scripts to databricks notebook

I'm looking to convert a Python .py file into a format compatible with Databricks, where the file can be split into multiple cells without using the Databricks GUI. Eventually, I’d like to automate this process. What’s the minimal code needed to create a…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,211 questions
asked 2024-10-29T14:45:29.6666667+00:00
Stephen-5002 40 Reputation points
accepted 2024-11-01T07:30:30.5633333+00:00
Stephen-5002 40 Reputation points
1 answer

Notebook : AccessDeniedException

Hello, I am trying to save a delta/parquet file into the ADLS Gen2 using a PySpark Notebook as follows : delta_output_path =…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,975 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,211 questions
asked 2024-10-29T15:28:46.6966667+00:00
Wajih Arfaoui 70 Reputation points
commented 2024-11-01T07:02:24.2333333+00:00
Smaran Thoomu 16,640 Reputation points Microsoft Vendor
1 answer

is there a spark connector for Azure Cosmos DB MongoDB API RU?

I'm using Azure Databricks to develop a proof of concept. Is there support for a Spark Connector for Azure Cosmos DB MongoDB API? If yes, what versions are supported. Is both batch and stream operations supported?

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,211 questions
asked 2024-10-30T16:53:46.0733333+00:00
Lum, Joey 0 Reputation points
commented 2024-11-01T06:56:29.9266667+00:00
Chandra Boorla 2,525 Reputation points Microsoft Vendor
1 answer

copy tables from sql server on-prem to .parquet format in delta lake

Currently our copy process for 10 tables is running in pipeline in ADF, our source is ON-PREM SQL Server and target is in BLOB Storage GEN2. it copies data(tables) in parquet file format in HOT storage Account --- StorageV2 (general purpose v2) but I…

Azure Storage Accounts
Azure Storage Accounts
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
3,216 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,211 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,808 questions
asked 2024-10-29T19:03:53.5233333+00:00
Zaeemul Haque 25 Reputation points
commented 2024-11-01T06:55:45.1833333+00:00
Chandra Boorla 2,525 Reputation points Microsoft Vendor
1 answer

I don't have access to admin console of databricks even though I am admin user

I created azure account and launched databricks, I don't have access to admin console of databricks even though I am the only admin user

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,211 questions
asked 2024-11-01T03:49:21.9833333+00:00
Ramya 0 Reputation points
answered 2024-11-01T05:20:34.03+00:00
Vinodh247 22,871 Reputation points
1 answer

Unable to access account console in daatabrics

Hello guys I created a storage account through my pay as u go subscription and lauched a premium tier workspace for databricks but i was trying to connect my workspace to metastore in unity catalog but i dont see my manage account button to access…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,211 questions
asked 2024-10-30T11:19:15.72+00:00
Mantri Sai Ashish 0 Reputation points
answered 2024-10-31T12:13:25.8766667+00:00
PRADEEPCHEEKATLA-MSFT 90,146 Reputation points Microsoft Employee
2 answers

I am not able to execute the "df.write.mode("overwrite").format('csv').option("path", "/mnt/sepadls/bronze/Employee").saveAsTable("Employee_Ext")

I am trying to create a folder called Employee in bronze layer and register as a table in hive meta store and below is the code. It's creating a folder called Employee in bronze layer but table is not getting registered and throwing an error as …

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,211 questions
asked 2024-10-26T13:31:43.7266667+00:00
Shivaraj 0 Reputation points
commented 2024-10-31T10:28:03.31+00:00
Vinodh247 22,871 Reputation points
2 answers

Complete Azure Databricks Cluster Recommendation

Hi, We are working on a project where we need to create Databricks cluster configurations recommendation with cluster versions, modes etc etc. Can anyone help ?

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,211 questions
asked 2024-10-28T07:18:47.36+00:00
Prasant Kumar Das 20 Reputation points
commented 2024-10-30T13:18:58.2966667+00:00
Smaran Thoomu 16,640 Reputation points Microsoft Vendor
1 answer One of the answers was accepted by the question author.

Connection Challenges with Azure Databricks and SQL Server On VM in Serverless compute.

We successfully configured VNet peering between the SQL VM's VNet and the Databricks VNet, allowing access to the SQL Server on VMs. Testing the connection from Databricks using a job cluster yielded successful results, confirming that everything…

SQL Server on Azure Virtual Machines
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,211 questions
asked 2024-10-28T05:19:04+00:00
Vetrivel Dhakshinamoorthy 20 Reputation points
accepted 2024-10-30T04:39:44.5666667+00:00
Vetrivel Dhakshinamoorthy 20 Reputation points
1 answer

not able to create a unity catalog

i want to create a unity catalog in azure databricks. manage account console is not accessible on azure databricks once i launch a workspace despite the "globle administrative role access" and premium subscription of azure…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,211 questions
Azure Role-based access control
Azure Role-based access control
An Azure service that provides fine-grained access management for Azure resources, enabling you to grant users only the rights they need to perform their jobs.
827 questions
Azure Data Catalog
Azure Data Catalog
An Azure service that serves as a system of registration and system of discovery for enterprise data assets.
103 questions
asked 2024-10-25T03:37:10.3533333+00:00
Giridhar Chandankhede 0 Reputation points
commented 2024-10-29T17:54:01.66+00:00
Chandra Boorla 2,525 Reputation points Microsoft Vendor
1 answer

Unable to Access Azure Databricks Admin Console as Global Administrator

I'm a Global Administrator in my Azure AD tenant, but I'm unable to access the Azure Databricks admin console for my workspace. Despite having full permissions across Azure AD, I’m encountering issues when trying to log in to manage Databricks. Has…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,211 questions
asked 2024-10-26T09:52:04.09+00:00
Mohan P V 5 Reputation points
commented 2024-10-29T11:15:05.95+00:00
Smaran Thoomu 16,640 Reputation points Microsoft Vendor
1 answer

worker and driver node

Hi, To process 100000 records how to calculate number of worker and driver nodes

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,211 questions
asked 2024-10-24T12:17:03.7133333+00:00
Vineet S 1,070 Reputation points
edited a comment 2024-10-29T10:53:43.7266667+00:00
Smaran Thoomu 16,640 Reputation points Microsoft Vendor
1 answer

Unable to Create Compute Clusters After Upgrading to Pay-As-You-Go

After upgrading from a free trial to a pay-as-you-go account, the ability to create compute clusters has been lost. The error message received is: "This account may not have enough CPU cores to start a cluster. Contact your administrator to increase…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,211 questions
asked 2024-10-25T21:32:43.4566667+00:00
glasier data 0 Reputation points
commented 2024-10-28T16:30:53.97+00:00
glasier data 0 Reputation points