Databricks SQL endpoint
Hi friends, where does the databricks SQL endpoint stand with comparison to other data warehousing technologies such as Synapse, snowflake, and google cloud? please provide metrics related comparison in terms of costs,scalability and performance. Which…
Why isn't code working as an expression for parameters in SQL Server Reporting with connection to Databricks SQL Warehouse using Simba Spark
Error: For more information about this error navigate to the report server on the local server machine, or enable remote errors ---------------------------- Query execution failed for dataset 'DataSet1'. (rsErrorExecutingCommand)…
How do I figure out what public IP ranges my Databricks workspace clusters are coming from?
Edit: I am rewriting this to clarify the ask. Relatively new to Databricks. I am trying to understand how outbound traffic from clusters is determined. It seems to differ if SCC is enabled vs when it's not. With no SCC: VMs start up with a…
Firewall Configuration for Custom Model Serving in Azure Databricks
Hi, I am encountering an error when trying to serve my custom LLM model endpoint. The error message reads: "Container image creation failed, see Build Logs for details. If there are no build logs, the failure may be due to storage firewall…
How to use a different version of a Spark Java library dependency (antlr4) in a Databricks notebook?
Hello. I need to use in a Databricks notebook a custom made Java library which depends on Drools v8.40.1.Final which depends on ANTLR4 v4.10.1. When I try to invoke a method in my Java library I get the following error: "ANTLR Tool version 4.10.1…
How to reduce unnecessary high memory usage in a Databricks cluster?
We are having unnecessary high memory usage even when nothing is running on the cluster. When the cluster first starts, it's fine, but when I run a script and it finishes executing, nothing gets back to the idle (initial) state (even hours after nothing…
Invalid records failed in DQ checks
We are capturing the records that failed in DQ checks by using Databricks in the Blob storage for business owners to resolve inconsistencies, we have added an extra column as DQ checks failed reason. I have the following: What if the particular record…
How to ignore the records in ADF Data Flows
Hi All I am building a data transamination using mapping data flows ,I have a time stamp field Like TimeStampUpdated in the target table. I want to lockup historical data with incremental data transamination and ignore the records coming in the…
How to parse nested json array of document in ADF data flow
Hi all I am trying to fitch the values from a nested josh array of document , I have used aggregate to convert into objects but not able to fitch the values of all child nodes like as below itOffer.item itOffer.item.SplOfr itOffer.item.buy …
Cluster Start-up Delayed. Please wait while we continue to try and start the cluster. No action is required from you. (cluster-id 0524-002352-kk357210-v2n)
Hello good people, I am getting this error "Cluster Start-up Delayed. Please wait while we continue to try and start the cluster. No action is required from you. (cluster-id 0524-002352-kk357210-v2n)" Please help. Thank You so much.
Azure Databricks workflow job failure
We have a stream workflow job that run 24*7 and loads the data in delta table for say: raw.deltaTableA Now, the problem is in case we are trying to optimize this delta (optimize raw.deltaTableA) table while the table is getting loaded we get frequent…
Delete the file from SharePoint location
Hi All, I am trying to copy the files from Share Point to ADLS and referring to the below URL pipeline to achieve the copy functionality. https://www.syntera.ch/blog/2022/10/10/copy-files-from-sharepoint-to-blob-storage-using-azure-data-factory/ I need…
Unable to downgrade Databricks workspace
When downgrading the Databricks Workspace, I receive the following message; However, none of the Enhanced Security options are currently enabled; Could you please help me identify the cause of the error?
How to setup modern Arcitechure for Small/Medium Business?
Currently we're using the following setup which is slow to process the data and is slow on the power bi side: Azure VM for third parties to upload via sftp C# script to ETL data to azure sql server and move files to ADLS Gen2 Power BI report pulling…
Databricks integration to Azure Synapse Serveless SQL Pool
Hi, What is the way to connect to Databricks to Azure Synapse serverless SQL pool when I am trying to connect using serverless SQL endpoint getting an error as - Py4JJavaError: An error occurred while calling o388.load. :…
CSV to XML conversion in databricks which have some blank values as well in csv
I am converting CSV data to xml and that CSV data has some blank values as well for a few columns let's take an example there are 4 columns in CSV and out of that for a row(record) 1 colom value is blank , so as an output in xml, I am getting a missing…
ADF | ADB Activity Execution Time on Job Clusters
Has anyone noticed adb notebooks running (on job clusters) faster in ADF ? we have sequential notebook activities and seeing the start up time of clusters to be as low as 2 minutes.
how to disable autoscaling local storage
I'm configuring the cluster with the 'enable_elastic_disk' parameter as 'false', using tfvars. ex: enable_elastic_disk = false. However, clustering in Databricks remains true. what to do?
Connecting Azure Databricks workspace to on-premises network - peering
I was following this tutorial to deploy a workspace for on prem database access. I created the VNET for Databricks as mentioned as well as the transit VNET. However, when I got to the option to peer the two VNETs the VNET peering option seems to be…
What is the difference between Databrick prepay and Databrick reservation in Azure ?
Hello, We are just considering ways to reduce Databrick cost in Azure other than buying RI for VMs behind Databrick clusters. What is the difference between Databrick prepay and Databrick reservation in Azure It seems Databrick reservation is named as…