How to reduce unnecessary high memory usage in a Databricks cluster?
We are having unnecessary high memory usage even when nothing is running on the cluster. When the cluster first starts, it's fine, but when I run a script and it finishes executing, nothing gets back to the idle (initial) state (even hours after nothing…
Azure Databricks
Azure | Azure Startups
Spark U/I is not loading for recently "terminated" job clustsers
I've been having trouble with a feature in Azure Databricks. The Spark U/I will not be shown for job clusters that have recently completed ("terminated"). There is a link to the Spark U/I in the databricks portal. But when you click the…
Azure Databricks

I am not able to start my new and first databricks cluster
I have upgraded my plan, and also tried to change the region from East US to East US 2 , but even then my cluster is not starting error 1: subscription issue error 2 : Message Cluster terminated.Reason: Cloud provider launch failure A cloud…
Azure Databricks
Remove characters in from values pyspark
Hi Expert, How to remove characters from column values pyspark sql I.e gffg546, gfg6544
Azure Data Lake Storage
Azure Synapse Analytics
Azure Databricks

Databricks-Connect also return module not found for multiple python files job.
Currently I'm connecting to databricks with local VS Code via databricks-connect. But my submmission all comes with error of module not found, which means the code in other python files not found. I tried: Move code into the folder with main.py …
Azure Databricks
Pyodbc install error in Azure Databricks: File not found error
Running the code below in Azure Databricks %sh curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add - curl https://packages.microsoft.com/config/ubuntu/16.04/prod.list > /etc/apt/sources.list.d/mssql-release.list apt-get…
Azure Databricks
Error when mounting Azure Data Lake Storage Gen1 account
I'm using the code below to mount gen1 account on azure databricks configs = {"fs.adl.oauth2.access.token.provider.type": "ClientCredential", "fs.adl.oauth2.client.id": <service-principal-appid>, …
Azure Data Lake Storage
Azure Databricks
how to resolve this error while getting data from databricks from power bi.Not able to load table in power bi
DataSource.Error: ODBC: ERROR [HY000] [Microsoft][DSI] (20039) Cannot store ""."".""."REMARKS" value in temporary table without truncation. (Column metadata implied a maximum of 512 bytes, while provided value is…
Azure Databricks
How do I connect to Databricks with .NET and run delta table queries
Is there a way to connect to Databricks and run Delta table queries from .NET? I've seen a few approaches using ODBC/JDBC but is there some other which excludes these types of connection? Maybe managed identity? If applicable, where can I find some…
Azure Databricks
Developer technologies | C#

Allow only a specific Azure Databricks instance to connect to keyvault
I want to connect to Azure Keyvault using "Azure Key Vault-backed scopes" in Databricks. Now on the Keyvault side, the way we allow an application to connect is via: RBAC - We are using the RBAC permission model. Add an identity in RBAC and…
Azure Databricks
how to fetch data from Azure Active Directory(AD) by using either ADF or databricks
To fetch data from Azure Active Directory (AD) using either Azure Data Factory (ADF) or Azure Databricks, Pleae let me know in detail. thanks
Azure Databricks
Azure Data Factory
Windows for business | Windows Client for IT Pros | Directory services | Active Directory
Adding datetime as dynamic content to a query
This is my query SELECT Id, ... FROM Task WHERE LastModifiedDate > @{activity('Lookup_SF_Task').output.firstRow.LastExtractRecordDate} This is how LastExtractRecordDate is being pulled by a Look Up module. This is the error…
Azure Databricks
Azure Data Factory
Developer technologies | Transact-SQL
Community Center | Not monitored
How to configure ADF pipeline run, linked service, so it uses Databricks serverless compute
Databricks has recently announced serverless compute for workflows: https://learn.microsoft.com/en-us/azure/databricks/workflows/jobs/run-serverless-jobs I would like to be able to execute Azure Data Factory (ADF) jobs using this…
Azure Databricks
Azure Data Factory
Azure data factory issue
Hi, We running some teste with Azure data factory and setup a Demo enviorment to copy some of the tabels of our DB to a Azure SQL db. we used the COpy data wizard to setup the ADF and connection to on-prem db and to Azure DB works. When try to copy…
Azure Databricks

How to disable Azure Databricks Personal Compute policy?
Hello, can you please provide details how to disable Azure Databricks Personal Compute policy for a) all users b) non-admin users Thanks!
Azure Databricks
com.databricks.sql.io.FileReadException: Error while reading file wasbs:REDACTED_LOCAL_PART@****.blob.core.windows.net/
Hi, I am getting the following error message: com.databricks.sql.io.FileReadException: Error while reading file…
Azure Blob Storage
Azure Databricks
extra SQL tables
Hi, Is it possible to add more tables to an existing datasets in Azure data factory? We have a pipeline that copy some on perm SQL tables to the Azure DB and everything works, but now I want to add some more tables to the data sets but dont see any…
Azure Databricks
Azure Databricks Synapse Connectivity
We are trying to use PolyBase in Azure Data Factory to copy the Delta lake table to Synapse. Using a simple Copy Activity in Azure Data Factory, our linked Services connections from Delta lake and Synapse show connection is successful, yet the copy step…
Azure Databricks

JDBC\ODBC connection to Azure Dataricks fails
Hi, I referred the following links in net and tried all, but still I am not connecting to Azure Databricks database https://docs.databricks.com/integrations/bi/power-bi.html https://docs.databricks.com/integrations/bi/jdbc-odbc-bi.html …
Azure Databricks
Databricks vnet peering
Environment: Single Azure subscription One resource group rgA, containing a vnet vnetA and some other resources I have contributor rights to this resource group. Done: I created a Databricks workspace myDatabricks. I…