2,129 questions with Azure Databricks tags

Sort by: Updated
0 answers

A vendor is trying to dump data in our azure blob storage and they are able to do it from the local environment but are not able to do it from their cloud platform

This is the error they are getting. I don't exactly know where the issue is.

Azure Storage Accounts
Azure Storage Accounts
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
3,053 questions
Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
2,759 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,129 questions
asked 2024-08-26T08:52:45.7733333+00:00
Shubhjeet Tiwari 0 Reputation points
commented 2024-08-27T09:40:12.5766667+00:00
Nehruji R 6,196 Reputation points Microsoft Vendor
1 answer One of the answers was accepted by the question author.

Databricks Error (An error occurred while calling o399.ls. : org.apache.hadoop.fs.FileAlreadyExistsException: Operation failed: "This endpoint does not support BlobStorageEvents or SoftDelete. Please disable these account features if you would like to us)

I have continously occur this error, An error occurred while calling o399.ls. : org.apache.hadoop.fs.FileAlreadyExistsException: Operation failed: "This endpoint does not support BlobStorageEvents or SoftDelete. Please disable these account features…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,129 questions
asked 2024-08-27T04:15:53.8366667+00:00
M. Abdullah Bin Aftab 40 Reputation points
commented 2024-08-27T07:08:34.1+00:00
PRADEEPCHEEKATLA-MSFT 87,731 Reputation points Microsoft Employee
0 answers

Support Needed for Accessing Azure Translator from Azure Databricks with Identity Authentication

I am looking for support in accessing Azure Translator from Azure Databricks using Identity Authentication. Currently, I am able to access it via an Access Key, but we want to transition to Identity Authentication.

Azure Translator
Azure Translator
An Azure service to easily conduct machine translation with a simple REST API call.
381 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,129 questions
asked 2024-08-12T14:45:07.7266667+00:00
Mahesh Sanga 0 Reputation points Microsoft Employee
commented 2024-08-27T07:03:16.9566667+00:00
YutongTie-MSFT 49,906 Reputation points
0 answers

Databricks runtime version 14.3 throwing error

Hello, I have multiple databricks job reporting failures when I use Databricks runtime version 14.3 (14.3.x-scala2.12) The same job passes if I use Databricks runtime version 12.2 Attached is the error seen in stderr.log of the cluster, also mentioned…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,129 questions
asked 2024-08-27T06:24:26.33+00:00
Amit Singh 60 Reputation points
0 answers

Databricks workspace deployment failed with BadRequest error

Databricks workspace creation and deployment failed with the following BadRequest error {"error":{"code":"BadRequest","message":"BAD_REQUEST: Account per user limit exceeded, each user is only allowed to join…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,129 questions
asked 2024-08-22T12:40:58.17+00:00
Mudit Surana 20 Reputation points
commented 2024-08-27T04:27:36.3033333+00:00
PRADEEPCHEEKATLA-MSFT 87,731 Reputation points Microsoft Employee
1 answer

Azure Databricks keeps notebooks in its cloud account free of cost?

I was reading the architecture of Databricks from the following video. And I came through this  If notebooks are stored in the control plane on the Databricks cloud account, then does that mean Databricks allocates storage for the client free of cost? I…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,129 questions
asked 2024-08-22T18:27:40.5233333+00:00
Dhruv Singla 85 Reputation points
commented 2024-08-27T04:12:25.93+00:00
PRADEEPCHEEKATLA-MSFT 87,731 Reputation points Microsoft Employee
1 answer

System deny assignment created by Azure Databricks

I need to create static IP for Databricks but I faced errors to create cloud NAT The client 'live.com#myemail' with object id 'objectid' has permission to perform action 'Microsoft.Resources/deployments/validate/action' on scope…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,129 questions
asked 2024-08-08T08:56:36.78+00:00
Vadym Urupa 0 Reputation points
commented 2024-08-27T03:25:36.3966667+00:00
PRADEEPCHEEKATLA-MSFT 87,731 Reputation points Microsoft Employee
0 answers

Databricks cluster stuck when trying to start in state of "Finding instances for new nodes"

Yesterday I had an issue with multiple notebooks attempting to run against an interactive cluster that they trigger to start but unfortunately the cluster stays in a spinning state saying it is "Finding instances for new nodes"... I even tried…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,129 questions
asked 2024-08-19T19:39:19.42+00:00
Daniel Woodhouse 0 Reputation points
commented 2024-08-26T20:38:19.7966667+00:00
Bhargava-MSFT 30,051 Reputation points Microsoft Employee
0 answers

Pipeline Failed due to error: InternalServerError executing request at adf

Around 15 triggers i have and only one dataset having trigger is failing everyday due to the error: InternalServerError executing request. The pipeline with attached triggers are published already. Any workaround for this issue will be more helpful.

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,798 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,129 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,443 questions
asked 2024-08-19T05:36:15.6033333+00:00
Lakshmi Moulya Nerella 0 Reputation points
commented 2024-08-26T16:34:22.5433333+00:00
phemanth 9,450 Reputation points Microsoft Vendor
0 answers

Can't login Databricks

I used the account role as owner, global administrator. And I checked the resource and Databricks workspace regions match. There is exactly one resource. The workspace you are trying to access does not exist in this Azure region, or your account…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,129 questions
asked 2024-08-25T03:19:36.2366667+00:00
Kelsey Chen 0 Reputation points
commented 2024-08-26T09:36:45.1866667+00:00
NIKHILANETHIKUNTA 1,110 Reputation points Microsoft Vendor
0 answers

Unable to login to Azure Databricks Account Console

I have a personal Azure pay-as-you-go subscription in which I have the 'Global Administrator' role. I am also the databricks account administrator. Since two weeks I am not able anymore to access the databricks account console.

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,129 questions
asked 2024-08-19T10:23:32.4833333+00:00
Markus 0 Reputation points
commented 2024-08-26T02:23:24.89+00:00
NIKHILANETHIKUNTA 1,110 Reputation points Microsoft Vendor
2 answers

I need to get data from HTTP APIs in Azure storage Gen 2 account. What should I use?

Basically I have HTTP API and I have GraphQL queries through which I can get the data. The limitations are, in one request, we can only get data for a month and only 2000 records at a time. For example if we have 10000 records for July 2024, then I need…

Azure Functions
Azure Functions
An Azure service that provides an event-driven serverless compute platform.
4,824 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,129 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,443 questions
asked 2024-08-06T11:52:22.38+00:00
Aqshata Akshay Ajnadkar 0 Reputation points
commented 2024-08-25T00:17:37.3966667+00:00
Vinodh247 15,701 Reputation points
2 answers One of the answers was accepted by the question author.

Unable to login to Azure Databricks Account Console

I have a personal Azure pay-as-you-go subscription in which I have the 'Global Administrator' role. I am also the databricks account administrator. Until two weeks ago, I was able to access the databricks account console without any issues, but I am…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,129 questions
asked 2024-08-12T09:01:10.54+00:00
Ramesh 50 Reputation points
commented 2024-08-23T16:44:41.3566667+00:00
Bhargava-MSFT 30,051 Reputation points Microsoft Employee
2 answers One of the answers was accepted by the question author.

Azure policy does not work for resources created by Azure Databricks

I'm working on using a policy to automatically assign the sku.name tag to virtual machines created by Databricks. These tags are important for my team's cost management analysis. However, I discovered in another discussion that resources marked as…

Azure Cost Management
Azure Cost Management
A Microsoft offering that enables tracking of cloud usage and expenditures for Azure and other cloud providers.
2,455 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,129 questions
Azure Policy
Azure Policy
An Azure service that is used to implement corporate governance and standards at scale for Azure resources.
854 questions
asked 2024-08-23T06:27:43.78+00:00
Michal Kalafarski 25 Reputation points
accepted 2024-08-23T11:02:45.0633333+00:00
Michal Kalafarski 25 Reputation points
1 answer One of the answers was accepted by the question author.

Data Factory monitoring by inserting data table

Hello, I would like to know the best way to insert Datafactory activity logs into my Databricks delta table, so that I can use dashbosrd and create monitoring in Databricks itself , can you help me? I would like every 5 minutes for all activity logs in…

Azure Monitor
Azure Monitor
An Azure service that is used to collect, analyze, and act on telemetry data from Azure and on-premises environments.
3,150 questions
Azure Storage Accounts
Azure Storage Accounts
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
3,053 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,129 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,443 questions
asked 2024-07-21T22:18:21.9966667+00:00
Hanna 220 Reputation points
commented 2024-08-23T10:30:19.33+00:00
Abdul 2,540 Reputation points Microsoft Vendor
2 answers One of the answers was accepted by the question author.

Data size of databricks delta tables

It has been observed that the size of delta tables are much less as compared to when checked the underlying delta files in the storage account. Suppose a databricks delta table raw.deltaTableA has size of 2MB if we check the size of underlying delta…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,451 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,129 questions
asked 2024-05-02T09:39:01.4133333+00:00
NIKHIL KUMAR 101 Reputation points
answered 2024-08-23T07:05:10.3+00:00
2 answers

while creating cluster in databricks i am getting following error Azure Quota Exceeded Exception: Error code: SkuNotAvailable, error message: The requested VM size for resource 'Following SKUs have failed for Capacity Restrictions: Standard_DS3_v2' is cur

while creating cluster in databricks i am getting following error Azure Quota Exceeded Exception: Error code: SkuNotAvailable, error message: The requested VM size for resource 'Following SKUs have failed for Capacity Restrictions: Standard_DS3_v2' is…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,129 questions
asked 2024-06-28T12:09:26.57+00:00
azurecloud 0 Reputation points
answered 2024-08-23T06:06:25.8033333+00:00
Amulya Chennaboyena 0 Reputation points
1 answer

Azure Databricks Workspace Unable to Access Workspace Storage Account

Databricks appears unable to access the workspace storage account. When attempting to display notebook cell results, an error returns: "Failed to upload command result to DBFS. Error message: dbstoragekbbxxxxxxxxx.dfs.core.windows.net: Name or…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,129 questions
asked 2024-08-22T16:10:55.7966667+00:00
Hutton, Randy 0 Reputation points
edited an answer 2024-08-22T21:40:40.13+00:00
Sina Salam 8,536 Reputation points
1 answer One of the answers was accepted by the question author.

CREATE TABLE does not overwrite location whereas CREATE OR REPLACE TABLE does

I am working on Azure Databricks, with Databricks Runtime version being - 14.3 LTS (includes Apache Spark 3.5.0, Scala 2.12). I am facing the following issue. Suppose I have a view named v1 and a database f1_processed created from the following…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,129 questions
asked 2024-02-08T07:15:40.59+00:00
Dhruv Singla 85 Reputation points
accepted 2024-08-22T17:11:05.01+00:00
Dhruv Singla 85 Reputation points
1 answer One of the answers was accepted by the question author.

Managed table overwrites existing location for delta but not for others

I am working on Azure Databricks, with Databricks Runtime version being - 14.3 LTS (includes Apache Spark 3.5.0, Scala 2.12). I am facing the following issue. Suppose I have a view named v1 and a database f1_processed created from the following…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,129 questions
asked 2024-02-08T09:11:34.64+00:00
Dhruv Singla 85 Reputation points
accepted 2024-08-22T17:10:15.4366667+00:00
Dhruv Singla 85 Reputation points