Azure Cost Management
A Microsoft offering that enables tracking of cloud usage and expenditures for Azure and other cloud providers.
3,041 questions
0 answers
Unexpected Azure Databricks VM Pricing – Matches RI Cost Without RI Purchase
I have an Azure Databricks customer with both Dev and Prod environments. The concern is regarding the VM cost associated with Databricks. The VM type is D4ads_v5 Linux, located in Southeast Asia. According to the Azure Pricing Calculator, the expected…
asked 2025-02-19T13:34:20+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(214.4, 67%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESM%3C/text%3E%3C/svg%3E)
Sundarakannan M
41
Reputation points
edited the question 2025-02-19T21:16:33.7933333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(12.8, 76%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EV%3C/text%3E%3C/svg%3E)
VarunTha
12,585
Reputation points Microsoft Vendor
2 answers
I’ve been using the following code for years to determine the latest table version using Databricks’ Time Travel feature without any issues. However, after adding a new row to the table recently, I encountered the following error:
Cannot time travel Delta table to version 1. Available versions: [3, 23]. This behavior is unexpected, as the code has worked reliably until now. Here is the code I’m using: from delta.tables import DeltaTable import pyspark.sql.functions dt =…
asked 2025-02-19T12:37:44.52+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(99.2, 32%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
Stephen01
125
Reputation points
answered 2025-02-19T18:59:48.6233333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(201.6, 18%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGG%3C/text%3E%3C/svg%3E)
Ganesh Gurram
4,170
Reputation points Microsoft Vendor
0 answers
How to set up secrets in Azure Databricks Cluster configs?
Hi there, I am currently working on setting up Spark configurations within our Databricks cluster to access Azure Data Lake Storage (ADLS) using OAuth and a service principal. Our goal is to configure these settings at the cluster level, so that we can…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(140.8, 63%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
1 answer
Automate Lineage into azure purview from Azure Databricks Unity Catalog
Hi Team, In my last question someone from Microsoft team suggested to create lineage tables separately and scan the databricks unity catalog source and then lineage automatically came to azure purview. But when I tried this solution, I'm not able to get…
asked 2025-02-13T08:15:02.3066667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(297.6, 5%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESL%3C/text%3E%3C/svg%3E)
Sri Lakshman Velugubantla
20
Reputation points
commented 2025-02-19T16:29:37.6133333+00:00
Sri Lakshman Velugubantla
20
Reputation points
2 answers
Unexpected Azure Databricks VM Pricing – Matches RI Cost Without RI Purchase
I have an Azure Databricks customer with both Dev and Prod environments. The concern is regarding the VM cost associated with Databricks. The VM type is D4ads_v5 Linux, located in Southeast Asia. According to the Azure Pricing Calculator, the expected…
asked 2025-02-19T13:20:43.8466667+00:00
Sundarakannan M
41
Reputation points
answered 2025-02-19T16:16:17.3133333+00:00
Vinodh247
28,211
Reputation points MVP
1 answer
Troubleshooting Azure Databricks to On-premise MS SQL Server VPN tunnel instability
We have a VPN tunnel that connects Azure Databricks to an on-premise server. The tunnel is however instable, and the phase 2 selector goes down daily at random times. We did an audit and it does seem to be mostly after working hours, varying from 7pm to…
asked 2025-02-14T10:35:45.9233333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(137.6, 26%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EN%3C/text%3E%3C/svg%3E)
Nathalie
20
Reputation points
commented 2025-02-19T10:10:54.7+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 15%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPB%3C/text%3E%3C/svg%3E)
Praveen Bandaru
430
Reputation points Microsoft Vendor
1 answer
Azure Databricks - Database Error
I created the following database in Azure Databricks - CREATE DATABASE IF NOT EXISTS f1_processed LOCATION "/mnt/<Container_Name>/<Folder_Name>" Database got created under "hive_metastore" Then I'm trying to create a…
asked 2025-02-13T04:55:01.34+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(131.20000000000002, 57.99999999999999%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERM%3C/text%3E%3C/svg%3E)
Ramanathan Murugan Veerabagu
0
Reputation points
commented 2025-02-18T12:50:28.17+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 8%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EST%3C/text%3E%3C/svg%3E)
Smaran Thoomu
20,295
Reputation points Microsoft Vendor
1 answer
Pyspark dataframe is taking too long to save on ADLS from Databricks.
I'm running a notebook on Azure databricks using a multinode cluster with 1 driver and 1-8 workers(each with 16 cores and 56 gb ram). Reading the source data from Azure ADLS which has 30K records. Notebook is consist of few transformation steps, also…
asked 2022-11-29T10:09:42.24+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(179.20000000000002, 57.00000000000001%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPR%3C/text%3E%3C/svg%3E)
Pratik Roy
1
Reputation point
commented 2025-02-18T04:26:33.81+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 36%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMT%3C/text%3E%3C/svg%3E)
Madhur Tyagi
0
Reputation points
2 answers
Can't access the Azure Databricks Accounts Console
Hi there, When I tried to log on to https://accounts.azuredatabricks.net/login, I encountered an error. It says: Selected user account does not exist in tenant 'Microsoft Services' and cannot access the application…
asked 2024-07-30T12:22:12.19+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(12.8, 39%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EZZ%3C/text%3E%3C/svg%3E)
Ziwei Zeng
10
Reputation points
commented 2025-02-17T16:22:40.1333333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(211.20000000000002, 78%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERK%3C/text%3E%3C/svg%3E)
Ramu Kamath
5
Reputation points
1 answer
Getting error while creating database
In Azure Databricks, I'm getting error while creating Database with the following syntax : - CREATE DATABASE IF NOT EXISTS <DB_Name> LOCATION "/mnt/<Container_Name>/<Folder_Name>" error is 1 "CREATE SCHEMA in Unity…
asked 2025-02-13T04:09:46.5566667+00:00
Ramanathan Murugan Veerabagu
0
Reputation points
commented 2025-02-17T12:16:32.76+00:00
Smaran Thoomu
20,295
Reputation points Microsoft Vendor
1 answer
Unable to Find 'Manage Accounts' Menu for Unity Catalog Metastore Setup
I'm trying to set up the Unity Catalog Metastore, but I can't find the 'Manage Accounts' menu in the 'Accounts' section, so I can't access the account console. Can anyone help me with this issue?"
asked 2025-02-10T18:33:16.54+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 96%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDS%3C/text%3E%3C/svg%3E)
deepak shetty
0
Reputation points
edited a comment 2025-02-12T15:41:47.8066667+00:00
Ganesh Gurram
4,170
Reputation points Microsoft Vendor
2 answers
Is Azure Synapse being discontinued?
Hello everyone! I recently heard that Microsoft might be discontinuing the development of Azure Synapse, and I’d like to know more about it if that’s the case. If it is being deprecated, is there an alternative to Serverless SQL Pool? I know Databricks…
asked 2025-02-10T12:10:09.0033333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(249.60000000000002, 40%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EES%3C/text%3E%3C/svg%3E)
EDUARDO SOARES AZZOLIN
0
Reputation points
edited the question 2025-02-12T10:49:31.9+00:00
Smaran Thoomu
20,295
Reputation points Microsoft Vendor
3 answers
How to specify a custom catalog name for Azure Databricks Delta Lake Dataset in ADF
Hello, I am creating an Azure Databricks Delta Lake Dataset in ADF and I am only able to choose the database name that links to Databricks's hive_metastore. How can I specify a custom catalog name that I created in Databricks instead of…
asked 2024-01-04T06:25:53.11+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(249.60000000000002, 17%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ETY%3C/text%3E%3C/svg%3E)
Tom Young
0
Reputation points
commented 2025-02-12T09:26:43.0333333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(12.8, 48%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMS%3C/text%3E%3C/svg%3E)
Milankumar Soni, Meetkumar
0
Reputation points
1 answer
Unable to connect to Azure Databricks repo with Azure DevOps server 2020
We are establishing the connection with Azure Databricks repo with the On-premise Azure DevOps Server 2020. We created a service principal and added to the databricks. we created a PAT from On-premise Azure DevOps server and added in the linked…
asked 2025-02-10T07:20:19.6933333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(233.6, 38%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECV%3C/text%3E%3C/svg%3E)
Chunduru, Vamsee Krishna
0
Reputation points
commented 2025-02-12T01:36:37.4066667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(99.2, 65%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECB%3C/text%3E%3C/svg%3E)
Chandra Boorla
8,470
Reputation points Microsoft Vendor
1 answer
Vnet Injected Databricks Workspace with IP ACL gives 403 while executing notebook using compute cluster
I have an IP access list enabled on my Vnet injected databricks workspace in Azure. I can access my workspace as my VPN ip is enabled on the workspace IP list. However, when I try executing my notebook by attaching a cluster, it gives below error- 403:…
asked 2025-02-07T08:42:14.1133333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(22.400000000000002, 64%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPP%3C/text%3E%3C/svg%3E)
Pratima Patil
30
Reputation points
commented 2025-02-11T12:16:51.06+00:00
Smaran Thoomu
20,295
Reputation points Microsoft Vendor
1 answer
Azure Databricks and Key vault key versioning
We are building a Data encryption solution and are using pyspark AES encrypt and AES Decrypt function to encrypt the PII columns in the Databricks tables on Unity catalog. We have created views on top of these tables and using the same encryption key we…
asked 2025-01-28T07:07:43.85+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(12.8, 34%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHK%3C/text%3E%3C/svg%3E)
Hemanth Kumar
0
Reputation points
commented 2025-02-11T09:21:50.0066667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(188.79999999999998, 26%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EN%3C/text%3E%3C/svg%3E)
Navya
15,470
Reputation points Microsoft Vendor
2 answers
How to use databricks ai to auto generate data definitions for all the tables in my database?
I know we can go to the catalog in databricks and generate data definitions for columns inside of our database using ai, but is there a way of automatically generating these definitions without have to manually generate them and click accept on every…
asked 2024-03-06T21:31:37.0066667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(0, 49%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBZ%3C/text%3E%3C/svg%3E)
Bradley, Zack
0
Reputation points
commented 2025-02-10T18:05:50.6533333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(201.6, 93%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMC%3C/text%3E%3C/svg%3E)
Mauricio Caro
0
Reputation points
2 answers
how to connect azure cache redis to azure databricks through service principal
I need to connect azure redis to databricks using service principal . but I am getting this error Failed to connect to Redis: invalid username-password pair
asked 2025-02-04T09:34:56.25+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(227.2, 80%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENP%3C/text%3E%3C/svg%3E)
Nageswara.Podapati
0
Reputation points
commented 2025-02-06T18:49:01.67+00:00
Oury Ba-MSFT
20,346
Reputation points Microsoft Employee
2 answers
Error with Create Table USING DELTA LOCATION in training exercise
In the exercise https://microsoftlearning.github.io/mslearn-databricks/Instructions/Exercises/03-Delta-lake-in-Azure-Databricks.html the line of code spark.sql("CREATE TABLE AdventureWorks.ProductsExternal USING DELTA LOCATION…
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,335 questions
Azure Training
Azure Training
Azure: A cloud computing platform and infrastructure for building, deploying and managing applications and services through a worldwide network of Microsoft-managed datacenters.Training: Instruction to develop new skills.
2,073 questions
asked 2024-05-01T13:00:09.32+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(6.4, 31%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJM%3C/text%3E%3C/svg%3E)
James Mitchell
10
Reputation points
commented 2025-02-06T09:52:25.67+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(156.8, 33%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EIT%3C/text%3E%3C/svg%3E)
Igor Trush
0
Reputation points
1 answer
One of the answers was accepted by the question author.
How to check exact IP address of ADB workspace env .
i just want to know my IP address of Azure data bricks workspace env, I have vnet and subnet configured. How can i get exact IP address of ADB workspace. As i have more than 100 NIC to check the exact IP. Is their any easiest way to get exact ip…
asked 2025-02-04T02:27:14.43+00:00
Aetemaad Hussain
40
Reputation points
commented 2025-02-05T20:34:41.37+00:00
Ganesh Gurram
4,170
Reputation points Microsoft Vendor