How to replace Data flow activity having transformations inside it in ADF with other activity.
I have Azure data factory pipeline which have data flow activity. Data flow activity points to source file in storage account gets data from it as a source then performs different transformations on data using conditional split, derived column, flatten…
primary key in adf pipeline
ADF package is failing due to Primarky and not null column available in target table due to which pipiline is failing how to handle such situation in adf.. Please prrimary key is only creating unique constraint
How to Correctly Pass and Use Boolean Values from Datafactory to Databricks Notebook
How can I correctly pass a Boolean value from Datafactory to a Databricks notebook and use it in conditional logic? I configured a pipeline in Datafactory that calls a Databricks notebook. I attempted to pass a Boolean parameter from Datafactory as a…
I am unable to create compute cluster in databricks from azure free trial subcription
I am unable to create compute cluster in databricks from azure free trial subcription. Failed to add 1 container to the compute. Will attempt retry: false. Reason: Azure Quota Exceeded Exception
whitelist the serverless data plane subnets in the cloud region
I am following below instructions from the documentation to whitelist the serverless data plane subnets in the cloud region of your Databricks workspace. But unable to find ARM resource ID of the serverless compute subnet details …
how to fetch data from Azure Active Directory(AD) by using either ADF or databricks
To fetch data from Azure Active Directory (AD) using either Azure Data Factory (ADF) or Azure Databricks, Pleae let me know in detail. thanks
How does ADF Linked Service for Azure Databricks aligns with Job Compute Policy defined inside Azure Databricks
This is a two-part Question. First Part Context: I have ADF which contains several data pipelines. Some pipeline also includes a databricks notebook as an activity. I have created Linked Service Azure Data Factory to facilitate the pipeline which…
create the dateid table in sql server
Hi , how to populate the dateid and date in dimtime table using script as below output upto 2029 excluding saturday and sunday create table table (dateid int,day varchar(23), dateid | day 20240601 | 2024 June 01 20290601 | 2029 June…
Getting the size of parquet files from azure blob storage
I have a blob container abcd The folder structure is like below: abcd/Folder1/Folder a, Folder b…..Folder z Inside a particular Folder a/v1/full/20230505/part12344.parquet Similarly Folder b/v1/full/20230505/part9385795.parquet Scenario is I need to get…
Azure Databricks Billing
I am confused about how the databricks service is billed under Azure. From documentation, it is said that Databricks is totally integrated with Azure billing: one bill for both Azure infrastructure (VM, storage, Network traffic, etc) and Databricks…
merge statement in 2 data frame
Hi , how to use merge statement in 2 dataframe df1=spark.sql("" sellect cole1,col2 from table1""") df2=spark.sql("" sellect cole1,col2 from table2""") expected results merge into table2 using tabl1 on…
merge in sql server table using dataframe
how to use merge statement in 2 dataframe df1=spark.sql("" sellect cole1,col2 from table1""") Table2 from sql server
DataBricks Unity Catalog Lineage
Hi, I'm looking for support on the Databricks Unity Catalog (on the data lineage). So I'm trying to establish lineage between 2 schemas (with 50 odd tables within each schema). Data for the first schema is fetched from source files (via ADF pipeline),…
![](https://techprofile.blob.core.windows.net/images/VfQFAmOikEWfBHko2XlWTA.png?8D7F33)
Guidance on how to use Service Principal with Certificate to Authorize for EventHub Stream Read
I found this documentation https://github.com/Azure/azure-event-hubs-spark/blob/master/docs/use-aad-authentication-to-connect-eventhubs.md online on how to use service principal with certificate to use spark stream read from EventHubs, I want to do this…
How to reduce unnecessary high memory usage in a Databricks cluster?
We are having unnecessary high memory usage even when nothing is running on the cluster. When the cluster first starts, it's fine, but when I run a script and it finishes executing, nothing gets back to the idle (initial) state (even hours after nothing…
While running SQL query in Azure Databricks workspace i.e. on SQL warehouse as well as on UC enabled shared cluster facing an SSL handshake error
Hello Team, We have UC enabled Azure databricks workspace, also the Public access and delta sharing is disabled on our workspace. So while running the below SQL query on SQL Warehouse as well as on UC enabled shared cluster, I am receiving an…
Azure Databricks to Cognitive Search connection timing out
I have a question on Azure Databricks and Azure Cognitive Search connectivity and hope this is the right forum to ask it. We are getting a timeout message when creating an index , "ServiceRequestError: (<urllib3.connection.HTTPSConnection object…
Databricks - unable to connect to ai search over private endpoint
We are getting a timeout message when creating an index , "ServiceRequestError: (<urllib3.connection.HTTPSConnection object at 0x7f081820e1a0>, 'Connection to xxxxx-aisearch-02.search.windows.net timed out. (connect timeout=300)')". I…
Managed storage account's compliance
Azure Databricks managed storage accounts need to have the key access disabled. But since these have deny assignment, I am able to see / influence the configuration. How do I make these storage accounts be green for this compliance?
Can we run delta live tables with free tier azure account?
While running the delta live tables pipeline in azure databricks I'm getting an error saying: QuotaExceeded, error message: Operation could not be completed as it results in exceeding approved Total Regional Cores quota. Additional details - Deployment…
![](https://techprofile.blob.core.windows.net/images/VfQFAmOikEWfBHko2XlWTA.png?8D7F33)