2,534 questions with Azure Databricks tags

Sort by: Answers
7 answers

How to reduce unnecessary high memory usage in a Databricks cluster?

We are having unnecessary high memory usage even when nothing is running on the cluster. When the cluster first starts, it's fine, but when I run a script and it finishes executing, nothing gets back to the idle (initial) state (even hours after nothing…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,534 questions
Azure | Azure Startups
asked 2024-05-08T08:58:46.4433333+00:00
Senad Hadzikic 20 Reputation points
commented 2025-04-14T20:54:35.5033333+00:00
Quan Sun 21 Reputation points
7 answers

Spark U/I is not loading for recently "terminated" job clustsers

I've been having trouble with a feature in Azure Databricks. The Spark U/I will not be shown for job clusters that have recently completed ("terminated"). There is a link to the Spark U/I in the databricks portal. But when you click the…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,534 questions
asked 2021-04-07T20:49:06.167+00:00
David Beavon 991 Reputation points
edited the question 2023-02-09T05:48:17.95+00:00
PRADEEPCHEEKATLA 90,651 Reputation points Moderator
6 answers

I am not able to start my new and first databricks cluster

I have upgraded my plan, and also tried to change the region from East US to East US 2 , but even then my cluster is not starting error 1: subscription issue error 2 : Message Cluster terminated.Reason: Cloud provider launch failure A cloud…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,534 questions
asked 2022-05-28T20:16:57.883+00:00
dev_pro 6 Reputation points
answered 2022-12-20T19:57:57.913+00:00
Juggling Boy 6 Reputation points
6 answers One of the answers was accepted by the question author.

Remove characters in from values pyspark

Hi Expert, How to remove characters from column values pyspark sql I.e gffg546, gfg6544

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,562 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,378 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,534 questions
asked 2022-08-18T18:32:32.387+00:00
Shambhu Rai 1,411 Reputation points
commented 2022-08-18T21:11:00.163+00:00
Dillon Silzer 57,831 Reputation points Volunteer Moderator
6 answers

Databricks-Connect also return module not found for multiple python files job.

Currently I'm connecting to databricks with local VS Code via databricks-connect. But my submmission all comes with error of module not found, which means the code in other python files not found. I tried: Move code into the folder with main.py …

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,534 questions
asked 2020-07-09T23:26:37.607+00:00
Dong Yuan 1 Reputation point
answered 2021-08-20T10:48:50.943+00:00
Michał Urban 1 Reputation point
5 answers One of the answers was accepted by the question author.

Pyodbc install error in Azure Databricks: File not found error

Running the code below in Azure Databricks %sh curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add - curl https://packages.microsoft.com/config/ubuntu/16.04/prod.list > /etc/apt/sources.list.d/mssql-release.list apt-get…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,534 questions
asked 2021-06-16T23:25:50.96+00:00
Chung, Joshua 151 Reputation points
accepted 2021-06-17T15:58:23.647+00:00
Chung, Joshua 151 Reputation points
5 answers One of the answers was accepted by the question author.

Error when mounting Azure Data Lake Storage Gen1 account

I'm using the code below to mount gen1 account on azure databricks configs = {"fs.adl.oauth2.access.token.provider.type": "ClientCredential", "fs.adl.oauth2.client.id": <service-principal-appid>, …

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,562 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,534 questions
asked 2021-03-31T15:54:31.93+00:00
Sunita 22 Reputation points Microsoft Employee
accepted 2021-04-10T03:08:22.703+00:00
Sunita 22 Reputation points Microsoft Employee
5 answers

how to resolve this error while getting data from databricks from power bi.Not able to load table in power bi

DataSource.Error: ODBC: ERROR [HY000] [Microsoft][DSI] (20039) Cannot store ""."".""."REMARKS" value in temporary table without truncation. (Column metadata implied a maximum of 512 bytes, while provided value is…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,534 questions
asked 2023-10-23T18:45:35.15+00:00
Pratibha Khare 30 Reputation points
commented 2023-12-21T07:54:25.25+00:00
Sadiq Rahimansa 10 Reputation points
5 answers

How do I connect to Databricks with .NET and run delta table queries

Is there a way to connect to Databricks and run Delta table queries from .NET? I've seen a few approaches using ODBC/JDBC but is there some other which excludes these types of connection? Maybe managed identity? If applicable, where can I find some…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,534 questions
Developer technologies | C#
asked 2021-11-24T09:41:17.26+00:00
Martin Kostadinov 36 Reputation points
answered 2023-04-24T15:37:43.24+00:00
Konstantinos Passadis 19,591 Reputation points MVP
5 answers

Allow only a specific Azure Databricks instance to connect to keyvault

I want to connect to Azure Keyvault using "Azure Key Vault-backed scopes" in Databricks. Now on the Keyvault side, the way we allow an application to connect is via: RBAC - We are using the RBAC permission model. Add an identity in RBAC and…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,534 questions
asked 2022-01-11T01:21:28.393+00:00
Nidhi Singh 6 Reputation points
answered 2025-03-14T11:21:01.1533333+00:00
Martyna Magdalena Sikorska 0 Reputation points
5 answers

how to fetch data from Azure Active Directory(AD) by using either ADF or databricks

To fetch data from Azure Active Directory (AD) using either Azure Data Factory (ADF) or Azure Databricks, Pleae let me know in detail. thanks

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,534 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,651 questions
Windows for business | Windows Client for IT Pros | Directory services | Active Directory
asked 2024-07-04T10:41:44.7933333+00:00
Lakshmi Narayana Sarma Bhamidipati 160 Reputation points
commented 2024-07-22T13:18:19.4833333+00:00
Lakshmi Narayana Sarma Bhamidipati 160 Reputation points
5 answers One of the answers was accepted by the question author.

Adding datetime as dynamic content to a query

This is my query SELECT Id, ... FROM Task WHERE LastModifiedDate > @{activity('Lookup_SF_Task').output.firstRow.LastExtractRecordDate} This is how LastExtractRecordDate is being pulled by a Look Up module. This is the error…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,534 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,651 questions
Developer technologies | Transact-SQL
Community Center | Not monitored
asked 2021-04-09T13:02:01.443+00:00
Bakar, Cem 281 Reputation points
commented 2021-04-12T16:11:44.407+00:00
MartinJaffer-MSFT 26,236 Reputation points
4 answers

How to configure ADF pipeline run, linked service, so it uses Databricks serverless compute

Databricks has recently announced serverless compute for workflows: https://learn.microsoft.com/en-us/azure/databricks/workflows/jobs/run-serverless-jobs I would like to be able to execute Azure Data Factory (ADF) jobs using this…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,534 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,651 questions
asked 2024-05-01T12:12:06.9033333+00:00
Krzysztof Przysowa 20 Reputation points
answered 2025-07-03T09:06:41.83+00:00
Krzysztof Przysowa 20 Reputation points
4 answers

Azure data factory issue

Hi, We running some teste with Azure data factory and setup a Demo enviorment to copy some of the tabels of our DB to a Azure SQL db. we used the COpy data wizard to setup the ADF and connection to on-prem db and to Azure DB works. When try to copy…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,534 questions
asked 2021-02-03T22:09:33.96+00:00
Shahin Mortazave 491 Reputation points
commented 2021-02-04T18:44:38.607+00:00
Kumara Prasanna 1 Reputation point
4 answers One of the answers was accepted by the question author.

How to disable Azure Databricks Personal Compute policy?

Hello, can you please provide details how to disable Azure Databricks Personal Compute policy for a) all users b) non-admin users Thanks!

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,534 questions
asked 2023-04-24T14:17:34.23+00:00
Anita A 110 Reputation points
answered 2025-03-07T14:48:58.82+00:00
Robert Golik (o365) 1 Reputation point
4 answers

com.databricks.sql.io.FileReadException: Error while reading file wasbs:REDACTED_LOCAL_PART@****.blob.core.windows.net/

Hi, I am getting the following error message: com.databricks.sql.io.FileReadException: Error while reading file…

Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
3,201 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,534 questions
asked 2021-01-29T20:32:13.813+00:00
Mayuri Kadam 81 Reputation points Microsoft Employee
answered 2021-04-13T12:00:41.21+00:00
Eugen Mirosch 1 Reputation point
4 answers One of the answers was accepted by the question author.

extra SQL tables

Hi, Is it possible to add more tables to an existing datasets in Azure data factory? We have a pipeline that copy some on perm SQL tables to the Azure DB and everything works, but now I want to add some more tables to the data sets but dont see any…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,534 questions
asked 2021-02-05T12:43:18.487+00:00
Shahin Mortazave 491 Reputation points
accepted 2021-03-10T11:37:56.237+00:00
Shahin Mortazave 491 Reputation points
4 answers One of the answers was accepted by the question author.

Azure Databricks Synapse Connectivity

We are trying to use PolyBase in Azure Data Factory to copy the Delta lake table to Synapse. Using a simple Copy Activity in Azure Data Factory, our linked Services connections from Delta lake and Synapse show connection is successful, yet the copy step…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,534 questions
asked 2021-03-29T20:30:55.797+00:00
Sahar Mostafa 26 Reputation points Microsoft Employee
answered 2021-05-26T16:30:02.147+00:00
brajesh jaishwal 26 Reputation points
4 answers

JDBC\ODBC connection to Azure Dataricks fails

Hi, I referred the following links in net and tried all, but still I am not connecting to Azure Databricks database https://docs.databricks.com/integrations/bi/power-bi.html https://docs.databricks.com/integrations/bi/jdbc-odbc-bi.html …

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,534 questions
asked 2020-10-19T12:22:43.827+00:00
Rajesh 46 Reputation points
commented 2024-09-30T11:32:59.8566667+00:00
Sai Kiran Bandi (LTIMINDTREE LIMITED) 0 Reputation points Microsoft External Staff
4 answers

Databricks vnet peering

Environment: Single Azure subscription One resource group rgA, containing a vnet vnetA and some other resources I have contributor rights to this resource group. Done: I created a Databricks workspace myDatabricks. I…

Azure Virtual Network
Azure Virtual Network
An Azure networking service that is used to provision private networks and optionally to connect to on-premises datacenters.
2,775 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,534 questions
asked 2021-11-01T17:27:40.277+00:00
Johan Van Noten 1 Reputation point
answered 2021-11-03T17:22:36.237+00:00
ChaitanyaNaykodi-MSFT 27,481 Reputation points Microsoft Employee Moderator