Informatica powercentre to Azure
Hi friends, how to migrate informatica powercentre with its all pipeline and data assets to Azure? What are important steps to map and consider? Will it be as is state or I need to refactor it in Azure ? If it is as is format then information cloud is…
Data Factory Data flow Error Cluster creation failed
I have a data factory pipeline that's been running for a year without any issues then suddenly we received an error Internal Server Error In Synapse batch operation: 'Cluster creation failed due to unexpected error. Please retry' This error came from a…
UNABLE TO CONNECT FROM DATABRICKS TO AZURE CONTAINER
Good morning, am unable to connect from my databricks notebook to one particular container in my storage account, as i am able to connect to other containers in the storage account. I did a comparison of the permissions they look the same, unmount and…
I have a dataframe in Synapse Apache spark pool and I want to create a table for that dataframe in my dedicated sql pool
I have a dataframe in Synapse Apache spark pool and I want to create a table for that dataframe in my dedicated sql pool.
How do I create a Azure Databricks Workspace that is not Unity Catalog Enabled?
Hello Everyone, As I notice that a Azure Databricks workspace creation by default is Unity Catalog enabled. How do I create a Azure Databricks workspace that is not Unity Catalog enabled. Thank you Prasanna
column name information in databricks
Hey, how to find particular column name information in datarbricks example SELECT s.name, t.name, c.name FROM sys.columns c JOIN sys.tables t ON c.object_id = t.object_id JOIN sys.schemas s ON t.schema_id = s.schema_id WHERE c.name LIKE…
Databricks column mapping
Hey, I got less in json files but there are more columns target table in databricks how to map if receive situation like this { "Generic": { "id": "33", "Products": [ { …
Unable to read data from Databricks using PowerBI
I have a report that reads data (direct query) from Azure Databricks. When I open it on a Windows notebook, I can edit it correctly and work with it. However, when I open the same file on a remote desktop, I'm unable to load the data from Databricks. In…
Azure Databricks Automatic Schema Evolution
Definition: I've created a Delta Live Table pipeline to handle the real-time data. Use Case: I want my delta live table pipeline to handle the schema changes whenever data comes with a different schema. Example Athletes.csv Schema …
json data file load in databricks
Hey, could you please explain how we can load below curly bracket files in databricks when there are multiple unwanted curly brackets are there apart from main data? in this case data is available in till Result only { "Generic": { …
filter latest LastModified date in array in adf
Hi team i have lastmodified date of all files inside folder and i need to filterout latest modifieddate using adf filter activity without using foreach and mapping dataflows so need solution on this. { "name":…
How to load delta files from ADLS to SQL in ADF
Hi have requirement to load delta files from adls container to sql but delta files are partitioning so load all latest partitioned files using adf how to achive this.
Databricks configuration setup
Hi friends, how would you decide the architecture of the number of executors and worker nodes needed for Databricks project's initial phase? What are the important criteria or evaluation methods and key factors that must be considered?
How to create databricks workspace with private endpoint
I'm facing a problem while creating a databricks workspace using private endpoint. I followed the…
spark catalog whitelisted for shared clusters with unityCatalog
I have recently initiated unity catalog for my workspace with the use of shared clusters, now I want to check if a data base exists using the spark.catalog.databaseExists() but the py4j error says the class is not whitelisted. The issue is when using the…
how to manage the data plane of Azure databricks
Normally, how to manage Azure databricks workspace and the resources, objects and services controlled by the workspace
Migration of azure databricks notebooks to azure ML workspace
I have plenty of notebooks in my ADB workspace , how can I move them to my AML workspace and start using them
Error "Invalid configuration value detected for fs.azure.account.keyInvalid configuration value detected for fs.azure.account.key" when listing files stored in an Azure Storage account using "dbutils.fs.ls"
Hi, I want to get a list of files stored in an Azure Storage account using "dbutils.fs.ls" command in Databricks. But, I get the following error. Failure to initialize configuration for storage account AAAAAA.dfs.core.windows.net: Invalid…
Failed to get MI access token.
I have 25 pipelines running but all are failing from yesterday due to this error: ErrorCode=AdlsGen2OperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=ADLS Gen2 operation failed for: Failed to get MI access token.…
Finding data patterns in a free text field
Hello Team, We have a free text field where in the user enters Name. The name can be of a person or a company. Sometimes it can be multiple names concatenated with "&". If its the name of a person, then the format is LastName, FirstName…