1,890 questions with Azure Databricks tags

Sort by: Updated
1 answer

Informatica powercentre to Azure

Hi friends, how to migrate informatica powercentre with its all pipeline and data assets to Azure? What are important steps to map and consider? Will it be as is state or I need to refactor it in Azure ? If it is as is format then information cloud is…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,890 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,345 questions
asked 2024-03-04T14:17:03.74+00:00
Anshal 1,626 Reputation points
answered 2024-03-06T01:13:54.9633333+00:00
Dillon Silzer 54,091 Reputation points
1 answer One of the answers was accepted by the question author.

Data Factory Data flow Error Cluster creation failed

I have a data factory pipeline that's been running for a year without any issues then suddenly we received an error Internal Server Error In Synapse batch operation: 'Cluster creation failed due to unexpected error. Please retry' This error came from a…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,890 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,345 questions
asked 2024-02-28T14:31:00.75+00:00
Colinn Calaguas 45 Reputation points
accepted 2024-03-05T20:27:37.2766667+00:00
Colinn Calaguas 45 Reputation points
1 answer One of the answers was accepted by the question author.

UNABLE TO CONNECT FROM DATABRICKS TO AZURE CONTAINER

Good morning, am unable to connect from my databricks notebook to one particular container in my storage account, as i am able to connect to other containers in the storage account. I did a comparison of the permissions they look the same, unmount and…

Azure Storage Accounts
Azure Storage Accounts
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
2,610 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,890 questions
asked 2024-03-04T07:45:15.01+00:00
Osamwonyi Alekwe 40 Reputation points
accepted 2024-03-04T14:45:16.13+00:00
Osamwonyi Alekwe 40 Reputation points
2 answers One of the answers was accepted by the question author.

I have a dataframe in Synapse Apache spark pool and I want to create a table for that dataframe in my dedicated sql pool

I have a dataframe in Synapse Apache spark pool and I want to create a table for that dataframe in my dedicated sql pool.

Azure SQL Database
Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,315 questions
Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
2,374 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,253 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,890 questions
asked 2024-03-01T04:20:45.7966667+00:00
Aditya Singh 105 Reputation points
accepted 2024-03-04T06:27:15.3966667+00:00
Aditya Singh 105 Reputation points
1 answer

How do I create a Azure Databricks Workspace that is not Unity Catalog Enabled?

Hello Everyone, As I notice that a Azure Databricks workspace creation by default is Unity Catalog enabled. How do I create a Azure Databricks workspace that is not Unity Catalog enabled. Thank you Prasanna

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,890 questions
asked 2024-03-02T22:44:08.7+00:00
Prasanna Sabbella 0 Reputation points
edited the question 2024-03-04T05:15:57.95+00:00
ManoharLakkoju 455 Reputation points Microsoft Vendor
1 answer

column name information in databricks

Hey, how to find particular column name information in datarbricks example SELECT s.name, t.name, c.name FROM sys.columns c JOIN sys.tables t ON c.object_id = t.object_id JOIN sys.schemas s ON t.schema_id = s.schema_id WHERE c.name LIKE…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,890 questions
asked 2024-03-03T16:24:08.76+00:00
Vineet S 145 Reputation points
commented 2024-03-04T00:29:14.36+00:00
Vineet S 145 Reputation points
0 answers

Databricks column mapping

Hey, I got less in json files but there are more columns target table in databricks how to map if receive situation like this {     "Generic": {         "id": "33",         "Products": [             {             …

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,890 questions
asked 2024-03-01T11:45:07.4466667+00:00
Vineet S 145 Reputation points
commented 2024-03-01T21:27:16.5333333+00:00
QuantumCache 20,021 Reputation points
1 answer

Unable to read data from Databricks using PowerBI

I have a report that reads data (direct query) from Azure Databricks. When I open it on a Windows notebook, I can edit it correctly and work with it. However, when I open the same file on a remote desktop, I'm unable to load the data from Databricks. In…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,890 questions
asked 2024-02-26T08:24:44.5366667+00:00
Atariah, Dror 0 Reputation points
commented 2024-03-01T19:32:56.75+00:00
BhargavaGunnam-MSFT 25,156 Reputation points Microsoft Employee
1 answer

Azure Databricks Automatic Schema Evolution

Definition: I've created a Delta Live Table pipeline to handle the real-time data. Use Case: I want my delta live table pipeline to handle the schema changes whenever data comes with a different schema. Example Athletes.csv Schema …

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,890 questions
asked 2024-02-27T10:23:22.34+00:00
Harshit Chandani 40 Reputation points
commented 2024-03-01T13:49:27.08+00:00
Vinodh247-1375 10,616 Reputation points
1 answer One of the answers was accepted by the question author.

json data file load in databricks

Hey, could you please explain how we can load below curly bracket files in databricks when there are multiple unwanted curly brackets are there apart from main data? in this case data is available in till Result only {     "Generic": {       …

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,890 questions
asked 2024-03-01T01:59:19.57+00:00
Vineet S 145 Reputation points
accepted 2024-03-01T10:40:58.8333333+00:00
Vineet S 145 Reputation points
0 answers

filter latest LastModified date in array in adf

Hi team i have lastmodified date of all files inside folder and i need to filterout latest modifieddate using adf filter activity without using foreach and mapping dataflows so need solution on this. { "name":…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,890 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,345 questions
asked 2024-02-26T14:34:00.51+00:00
Prapul Kumar Dongari 116 Reputation points
commented 2024-02-29T19:34:36.3666667+00:00
BhargavaGunnam-MSFT 25,156 Reputation points Microsoft Employee
0 answers

How to load delta files from ADLS to SQL in ADF

Hi have requirement to load delta files from adls container to sql but delta files are partitioning so load all latest partitioned files using adf how to achive this.

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,315 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,890 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,345 questions
asked 2024-02-27T05:20:34.0166667+00:00
Prapul Kumar Dongari 116 Reputation points
commented 2024-02-29T10:27:29.6066667+00:00
Suba Balaji 11,186 Reputation points
1 answer One of the answers was accepted by the question author.

Databricks configuration setup

Hi friends, how would you decide the architecture of the number of executors and worker nodes needed for Databricks project's initial phase? What are the important criteria or evaluation methods and key factors that must be considered?

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,890 questions
asked 2024-02-27T12:08:08.3366667+00:00
Anshal 1,626 Reputation points
accepted 2024-02-29T07:25:12.8133333+00:00
Anshal 1,626 Reputation points
1 answer One of the answers was accepted by the question author.

How to create databricks workspace with private endpoint

I'm facing a problem while creating a databricks workspace using private endpoint. I followed the…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,890 questions
asked 2023-05-02T21:16:02.3033333+00:00
João Bálico 20 Reputation points
commented 2024-02-28T22:01:01.49+00:00
Alessandro Cardoso 5 Reputation points Microsoft Employee
0 answers

spark catalog whitelisted for shared clusters with unityCatalog

I have recently initiated unity catalog for my workspace with the use of shared clusters, now I want to check if a data base exists using the spark.catalog.databaseExists() but the py4j error says the class is not whitelisted. The issue is when using the…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,890 questions
asked 2023-05-15T06:37:13.7866667+00:00
Chandra Doredla 25 Reputation points
commented 2024-02-28T17:09:21.45+00:00
Syed Akbar 0 Reputation points
1 answer

how to manage the data plane of Azure databricks

Normally, how to manage Azure databricks workspace and the resources, objects and services controlled by the workspace

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,890 questions
asked 2024-02-27T19:05:49.3766667+00:00
Chen, Albert G 21 Reputation points
answered 2024-02-27T21:43:16.41+00:00
QuantumCache 20,021 Reputation points
1 answer

Migration of azure databricks notebooks to azure ML workspace

I have plenty of notebooks in my ADB workspace , how can I move them to my AML workspace and start using them

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,890 questions
asked 2024-02-19T05:04:46.0866667+00:00
KEERTHANA JAYADEVAN 66 Reputation points
commented 2024-02-27T11:06:41.3466667+00:00
romungi-MSFT 41,736 Reputation points Microsoft Employee
2 answers One of the answers was accepted by the question author.

Error "Invalid configuration value detected for fs.azure.account.keyInvalid configuration value detected for fs.azure.account.key" when listing files stored in an Azure Storage account using "dbutils.fs.ls"

Hi, I want to get a list of files stored in an Azure Storage account using "dbutils.fs.ls" command in Databricks. But, I get the following error. Failure to initialize configuration for storage account AAAAAA.dfs.core.windows.net: Invalid…

Azure Storage Accounts
Azure Storage Accounts
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
2,610 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,890 questions
asked 2023-06-21T11:09:15.6666667+00:00
Mohammad Saber 586 Reputation points
answered 2024-02-27T01:11:37.9033333+00:00
Aman Sehgal 0 Reputation points
1 answer

Failed to get MI access token.

I have 25 pipelines running but all are failing from yesterday due to this error: ErrorCode=AdlsGen2OperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=ADLS Gen2 operation failed for: Failed to get MI access token.…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,890 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,345 questions
Azure App Service
Azure App Service
Azure App Service is a service used to create and deploy scalable, mission-critical web apps.
6,695 questions
asked 2024-02-21T12:59:58.12+00:00
SHARAD SONWANE 0 Reputation points
commented 2024-02-26T21:36:03.2233333+00:00
QuantumCache 20,021 Reputation points
2 answers

Finding data patterns in a free text field

Hello Team, We have a free text field where in the user enters Name. The name can be of a person or a company. Sometimes it can be multiple names concatenated with "&". If its the name of a person, then the format is LastName, FirstName…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,890 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,345 questions
asked 2024-02-20T12:27:17.7766667+00:00
Nagesh CL 601 Reputation points
edited the question 2024-02-26T19:59:31.59+00:00
BhargavaGunnam-MSFT 25,156 Reputation points Microsoft Employee