2,029 questions with Azure Databricks tags

Sort by: Updated
1 answer

How to know which type of service we need within azure databricks for our implementation

I need to know the different types of services available within databricks to implement my solution more cost efficiently. Is there any resource from azure we can reach out to whom we can explain our implementation and they can provide the list of…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,029 questions
asked 2024-06-21T09:22:35.21+00:00
Arun Kumar 0 Reputation points
answered 2024-06-21T11:38:34.17+00:00
Smaran Thoomu 11,695 Reputation points Microsoft Vendor
1 answer

Error while implementing File Notification Mode: com.microsoft.azure.storage.StorageException: This request is not authorized to perform this operation using this permission.

While implementing File Notification Mode in Autoloader, I get the following error. Has anyone faced the similar issue? Note: The Databricks Service Principal is having Contributor role to Storage account. com.microsoft.azure.storage.StorageException:…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,029 questions
asked 2024-06-21T06:16:38.9166667+00:00
Hiran Amarathunga 25 Reputation points
commented 2024-06-21T10:27:22.34+00:00
Hiran Amarathunga 25 Reputation points
1 answer

Pipeline Executing Databricks Notebook Successfully Despite Stopped Cluster

In Azure Data Factory (ADF), I have a pipeline that executes a notebook in Azure Databricks. I noticed that even when the Databricks cluster is stopped, the ADF pipeline still completes successfully, and the notebook runs without any issues. Is this…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,029 questions
asked 2024-06-20T11:17:32.34+00:00
vikranth-0706 160 Reputation points
commented 2024-06-21T09:01:04.8933333+00:00
Smaran Thoomu 11,695 Reputation points Microsoft Vendor
1 answer

logic app to save data in adls gen2

Hi all, I would like to utilize logic app to write into adls gen2. After the file is saved in adls gen2, it should be able to read by databrick as delta table. May I know is there a specific file format to do it or how it can be achived? thanks

Azure Logic Apps
Azure Logic Apps
An Azure service that automates the access and use of data across clouds without writing code.
2,946 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,029 questions
asked 2024-06-20T15:02:05.7333333+00:00
Yang Chow Mun 101 Reputation points
commented 2024-06-21T08:51:16.0133333+00:00
Luis Arias 5,446 Reputation points
1 answer

vm cpu utilization

Hi, have 100 VMs in Azure portal's resource group for which i am running VM memory metrix(cpu usage) .. how can it will automatically recognized the new subscription of vm came so that it will show the cpu usage via loading adf or databricks pipeline

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,565 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,029 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,927 questions
Azure App Service
Azure App Service
Azure App Service is a service used to create and deploy scalable, mission-critical web apps.
7,198 questions
asked 2024-06-20T10:33:21.4333333+00:00
Vineet S 225 Reputation points
commented 2024-06-21T08:01:33.3766667+00:00
Smaran Thoomu 11,695 Reputation points Microsoft Vendor
2 answers

Managed storage account's compliance

Azure Databricks managed storage accounts need to have the key access disabled. But since these have deny assignment, I am able to see / influence the configuration. How do I make these storage accounts be green for this compliance?

Azure Storage Accounts
Azure Storage Accounts
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
2,833 questions
Azure Managed Applications
Azure Managed Applications
An Azure service that enables managed service providers, independent software vendors, and enterprise IT teams to deliver turnkey solutions through the Azure Marketplace or service catalog.
117 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,029 questions
asked 2024-06-19T17:48:49.89+00:00
Chitra Gurumurthy (NON EA SC ALT) 0 Reputation points Microsoft Employee
answered 2024-06-20T10:50:03.1366667+00:00
Nehruji R 3,801 Reputation points Microsoft Vendor
0 answers

DataBricks Unity Catalog Lineage

Hi, I'm looking for support on the Databricks Unity Catalog (on the data lineage). So I'm trying to establish lineage between 2 schemas (with 50 odd tables within each schema). Data for the first schema is fetched from source files (via ADF pipeline),…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,029 questions
asked 2024-06-20T06:15:49.5666667+00:00
S, Santhosh M 0 Reputation points
commented 2024-06-20T08:48:00.6633333+00:00
PRADEEPCHEEKATLA-MSFT 83,301 Reputation points Microsoft Employee
0 answers

How to solve Invalid SessionHandle error with Azure Databricks ?

I am applying a SQLDatabaseChain Chatbot model by using LangChain SQLDatabaseChain and GPT4. I first created this model on Azure Databricks notebook like this : import json import os import langchain import mlflow from mlflow.models import…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,029 questions
asked 2024-06-17T14:38:14.4333333+00:00
Feng YIJUN 0 Reputation points
commented 2024-06-20T07:55:38.4+00:00
Feng YIJUN 0 Reputation points
1 answer

Cross tenant AAD authentication for Purview & Databricks

Hi, I want to know if for Purview & Databricks, is it possible to authenticate with cross tenant AAD? That is to say, can users belonging to AAD in tenant1, be able to login to Purview & Databricks which are setup in tenant2? Thanks

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,029 questions
Active Directory
Active Directory
A set of directory-based technologies included in Windows Server.
6,091 questions
Microsoft Purview
Microsoft Purview
A Microsoft data governance service that helps manage and govern on-premises, multicloud, and software-as-a-service data. Previously known as Azure Purview.
1,010 questions
asked 2024-06-20T03:44:04.3166667+00:00
Amit Singh 40 Reputation points
answered 2024-06-20T07:20:03.3966667+00:00
PRADEEPCHEEKATLA-MSFT 83,301 Reputation points Microsoft Employee
1 answer

How can I migrate data bricks from one subscription to another subscription during cross subscription migration ?

I am unable to move data brick from the source subscription to the destination subscription during cross-subscription migration

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,029 questions
asked 2024-06-19T19:20:06.0166667+00:00
Maaz Ahmed Nagori 0 Reputation points
answered 2024-06-20T05:42:31.93+00:00
PRADEEPCHEEKATLA-MSFT 83,301 Reputation points Microsoft Employee
0 answers

I am unable to move data brick from the source subscription to the destination subscription during cross-subscription migration

I am unable to move data brick from the source subscription to the destination subscription during cross-subscription migration

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,029 questions
asked 2024-06-19T19:22:30.1166667+00:00
Maaz Ahmed Nagori 0 Reputation points
commented 2024-06-19T20:16:58+00:00
BhargavaGunnam-MSFT 28,271 Reputation points Microsoft Employee
0 answers

How can I clear the cache in my Databricks Cluster

Ive tried many articles and links without success. Has anyone managed to successfully clear the cache on their Databricks cluster? This causes many issues as my memory is near its peak, meaning my notebooks often crashes. Any help or advised would be…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,029 questions
asked 2024-06-18T11:53:11.7766667+00:00
richardthomas66@hotmail.com 0 Reputation points
commented 2024-06-19T19:35:52.0033333+00:00
BhargavaGunnam-MSFT 28,271 Reputation points Microsoft Employee
0 answers

While running databricks migrate pipeline facing an issue with invalid configuration for storage account key

Hello Team, I am trying to run the yaml pipeline for azure databricks migration from Non UC workspace to UC workspace for reference this is the rep https://github.com/databrickslabs/migrate so while exporting the hive metastore, I am running into error…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,398 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,029 questions
asked 2024-06-17T15:46:25.1033333+00:00
Ashwini Gaikwad 110 Reputation points
commented 2024-06-19T19:35:27.0533333+00:00
BhargavaGunnam-MSFT 28,271 Reputation points Microsoft Employee
0 answers

Restricting files/folders to upload into External volumes in Azure databricks UC workspace

Hello Team, Is there a way to restrict the files or folders to upload/download from external volumes same like DBFS? Is there any option to disable the uploading files/folders feature in external volumes of azure databricks workspace with Unity Catalog.…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,029 questions
asked 2024-06-06T17:37:57.5833333+00:00
Ashwini Gaikwad 110 Reputation points
commented 2024-06-19T17:44:17.9266667+00:00
BhargavaGunnam-MSFT 28,271 Reputation points Microsoft Employee
1 answer One of the answers was accepted by the question author.

How to restore myspace and files created with Student subscription?

Hello, I had a Student Azure subscription in 2022-2023 and created some resources and lots of files in Databricks. then my subscription expired and I could not renew it though I am still a student. I created a paid subscription today in hope to see my…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,029 questions
asked 2024-06-19T08:48:44.8133333+00:00
Maria 71 Reputation points
accepted 2024-06-19T08:57:47.1733333+00:00
Maria 71 Reputation points
2 answers

Databricks not sending audit logs to event hub

Hi, I'm trying to push all the logs from databricks using the diagnostic tool to event hub but is not working, it didn't push anything. I'm using the root access policy and also already created the eventhub name, what else I'm probably missing? Thanks in…

Azure Event Hubs
Azure Event Hubs
An Azure real-time data ingestion service.
581 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,029 questions
asked 2024-06-13T19:41:27.85+00:00
Danny AMAYA 0 Reputation points
commented 2024-06-19T04:22:15.28+00:00
PRADEEPCHEEKATLA-MSFT 83,301 Reputation points Microsoft Employee
2 answers One of the answers was accepted by the question author.

How to setup modern Arcitechure for Small/Medium Business?

Currently we're using the following setup which is slow to process the data and is slow on the power bi side: Azure VM for third parties to upload via sftp C# script to ETL data to azure sql server and move files to ADLS Gen2 Power BI report pulling…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,398 questions
Azure Virtual Machines
Azure Virtual Machines
An Azure service that is used to provision Windows and Linux virtual machines.
7,407 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,029 questions
asked 2024-05-23T20:55:59.0633333+00:00
Jordan 25 Reputation points
accepted 2024-06-18T22:51:22.4966667+00:00
Jordan 25 Reputation points
1 answer One of the answers was accepted by the question author.

How to get the public IP range of my Azure Databricks cluster? I Need to whitelist this IP range with a host that I want to connect and get data from.

How to get the public IP range of my Azure Databricks cluster? I Need to whitelist this IP range with a host that I want to connect to and get data from.

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,029 questions
asked 2024-06-13T08:19:20.98+00:00
Makarand Nidhalkar 20 Reputation points
accepted 2024-06-18T13:41:17.45+00:00
Makarand Nidhalkar 20 Reputation points
3 answers

How to reduce unnecessary high memory usage in a Databricks cluster?

We are having unnecessary high memory usage even when nothing is running on the cluster. When the cluster first starts, it's fine, but when I run a script and it finishes executing, nothing gets back to the idle (initial) state (even hours after nothing…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,029 questions
asked 2024-05-08T08:58:46.4433333+00:00
Senad Hadzikic 20 Reputation points
answered 2024-06-18T07:47:33.34+00:00
Luigi Greselin 0 Reputation points
1 answer One of the answers was accepted by the question author.

Databricks Simba Spark ODBC .NET8 C# Driver Parameters in SQL Queries

Hello, I'm using Simba ODBC driver v2.8.0 in order to query data from my azure databrick sql warehouse into a .net 8 Asp.net Api App. The ODBC driver works fine using plain text query but i need to parametrize the query. Searching around I found that it…

ASP.NET Core
ASP.NET Core
A set of technologies in the .NET Framework for building web applications and XML web services.
4,307 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,029 questions
asked 2024-06-03T15:30:12.4666667+00:00
Luigi Navarra 25 Reputation points
accepted 2024-06-18T07:16:04.8966667+00:00
Luigi Navarra 25 Reputation points