Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,364 questions
1 answer
Azure to AWS
Hello We need to transfer files from ADLS to AWS (S3 bucket) for a SAS application hosted in third party in batches. We need to ensure data security and best practices. My understanding, we can use ADF to create a linked service for AWS S3 but IT DOES…
asked 2024-05-20T08:39:35.8366667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(102.4, 35%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
Sourav
80
Reputation points
answered 2024-05-20T11:47:58.1733333+00:00
Amira Bedhiafi
16,071
Reputation points
1 answer
CSV to XML conversion in databricks which have some blank values as well in csv
I am converting CSV data to xml and that CSV data has some blank values as well for a few columns let's take an example there are 4 columns in CSV and out of that for a row(record) 1 colom value is blank , so as an output in xml, I am getting a missing…
asked 2024-05-16T08:34:02.7366667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(291.2, 75%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EM%3C/text%3E%3C/svg%3E)
Manoj
0
Reputation points
commented 2024-05-20T10:40:36.41+00:00
ShaikMaheer-MSFT
38,126
Reputation points Microsoft Employee
0 answers
Unable to Get $100 Free Credit with Azure for Students Plan
Hi everyone, I'm having trouble accessing the $100 free credit that comes with the Azure for Students plan. Here are the details of my situation: I'm a student at DY Patil International University. I have a verified GitHub Student Developer Pack…
asked 2024-05-20T10:26:41.02+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(67.2, 77%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPP%3C/text%3E%3C/svg%3E)
Parth Patil
0
Reputation points
asked 2024-05-20T10:26:41.02+00:00
Parth Patil
0
Reputation points
0 answers
py4j.security.Py4JSecurityException
Hello I am trying to run spark XGBoostRegression model on Databricks cluster with Databricks runtime: 14.3 LTS. I am getting the following error: Py4JError: An error occurred while calling o547.resourceProfileManager. Trace:…
asked 2024-05-06T12:48:28.3166667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(67.2, 16%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAR%3C/text%3E%3C/svg%3E)
Ahuja, Rachit
0
Reputation points
edited a comment 2024-05-20T08:05:59.71+00:00
PRADEEPCHEEKATLA-MSFT
78,986
Reputation points Microsoft Employee
0 answers
Why create compute is taking long time?
I am trying create a compute for my workspaces i tried every combination still it is not working
asked 2024-05-03T13:41:31+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(300.8, 13%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAP%3C/text%3E%3C/svg%3E)
Aditya Parida
0
Reputation points
commented 2024-05-20T07:45:45.62+00:00
PRADEEPCHEEKATLA-MSFT
78,986
Reputation points Microsoft Employee
1 answer
Azure Databricks fail to install Geospark libraries from Maven
Hi Team , I am attempting to add below two geospark Maven libraries to my Azure Databricks interactive cluster with Runtime Version 14.3 LTS . However , I am getting below error Library installation attempted on the driver node of cluster…
asked 2024-04-15T06:24:17.8033333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(96, 57.99999999999999%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAS%3C/text%3E%3C/svg%3E)
Anuj, Singh (Cognizant)
25
Reputation points
commented 2024-05-20T06:03:18.5966667+00:00
PRADEEPCHEEKATLA-MSFT
78,986
Reputation points Microsoft Employee
1 answer
How to ship Azure Databricks artifacts from Dev->QA->Prod through Azure Devops Pipelines?
We have a Azure Databricks workspace and Dev/QA/Prod environments. Everytime the Data engineers have to ship the artifacts from nonprod -> prod (e.g. python notebooks, config modules, etc) they have to copy the artifacts manually over to the next…
asked 2024-04-29T21:17:03.8233333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 89%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EC%3C/text%3E%3C/svg%3E)
Cataster
641
Reputation points
commented 2024-05-20T05:59:43.13+00:00
PRADEEPCHEEKATLA-MSFT
78,986
Reputation points Microsoft Employee
1 answer
How do I add an inbound security rule if there is an default DenyAllInbound Rule that causes an error when attempting to create an inbound rule?
|Received an email with: The public IP address range for the Azure Databricks control plane will be updated on 30 May 2024—you may need to take action You're receiving this email because you use Azure Databricks. To support infrastructure …
asked 2024-04-30T17:39:32.7466667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(64, 18%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPS%3C/text%3E%3C/svg%3E)
Parris Sikorski (ALLEGIS GROUP HOLDINGS INC)
0
Reputation points Microsoft Vendor
commented 2024-05-20T05:57:54.2866667+00:00
PRADEEPCHEEKATLA-MSFT
78,986
Reputation points Microsoft Employee
0 answers
Custom libraries (wheel) for ADF Databricks Python activity run on serverless compute
I want to be able to execute Python scripts (via Databricks Python) from Azure Data Factory using serverless compute. Serverless compute does not support cluster level (compute scoped) libraries. In databricks workflows, it is being done as…
asked 2024-05-01T12:30:52.2366667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 31%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKP%3C/text%3E%3C/svg%3E)
Krzysztof Przysowa
0
Reputation points
commented 2024-05-20T05:56:04.02+00:00
PRADEEPCHEEKATLA-MSFT
78,986
Reputation points Microsoft Employee
1 answer
Error with Create Table USING DELTA LOCATION in training exercise
In the exercise https://microsoftlearning.github.io/mslearn-databricks/Instructions/Exercises/03-Delta-lake-in-Azure-Databricks.html the line of code spark.sql("CREATE TABLE AdventureWorks.ProductsExternal USING DELTA LOCATION…
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,957 questions
Azure Training
Azure Training
Azure: A cloud computing platform and infrastructure for building, deploying and managing applications and services through a worldwide network of Microsoft-managed datacenters.Training: Instruction to develop new skills.
1,017 questions
asked 2024-05-01T13:00:09.32+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(6.4, 31%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJM%3C/text%3E%3C/svg%3E)
James Mitchell
0
Reputation points
commented 2024-05-20T05:53:37.2866667+00:00
PRADEEPCHEEKATLA-MSFT
78,986
Reputation points Microsoft Employee
1 answer
PowerBI / Databrick can we edit data in report
When we create reports in PowerBi or in Databricks. can we edit the data in report and if it can updated in backend datasource. Please let me know if this possible
asked 2024-05-06T20:49:03.1766667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(83.2, 43%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPS%3C/text%3E%3C/svg%3E)
Pothiraj, Saranya-ADM
0
Reputation points
commented 2024-05-20T05:50:58.8333333+00:00
PRADEEPCHEEKATLA-MSFT
78,986
Reputation points Microsoft Employee
1 answer
How do I figure out what public IP ranges my Databricks workspace clusters are coming from?
Relatively new to Databricks. I have an existing workspace that was created years ago. It is vnet-injected but it has secured cluster connectivity (SCC) disabled. I need to know the outbound IP addresses/ranges the clusters would communicate on to…
asked 2024-05-08T22:13:53.72+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(185.6, 45%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
McDonald, Matthew
101
Reputation points
answered 2024-05-20T05:42:40.1833333+00:00
PRADEEPCHEEKATLA-MSFT
78,986
Reputation points Microsoft Employee
0 answers
Indexing a Pyspark dataframe
Hey guys, I am having a very large dataset as multiple parquets (like around 20,000 small files) which I am reading into a pyspark dataframe. I want to add an index column in this dataframe and then do some data profiling and data quality check…
asked 2024-05-09T07:29:38.0266667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(12.8, 81%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVS%3C/text%3E%3C/svg%3E)
Varun S Kumar
50
Reputation points
commented 2024-05-20T05:40:59.8933333+00:00
PRADEEPCHEEKATLA-MSFT
78,986
Reputation points Microsoft Employee
2 answers
Clusters are failing to launch. Cluster launch will be retried.
Hi, I am a newbie. Can someone show me how I can fix the below please? Details for the latest failure: Error: Error code: QuotaExceeded, error message: Operation could not be completed as it results in exceeding approved standardEDSv4Family Cores…
asked 2024-05-13T21:53:30.17+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(86.4, 18%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBC%3C/text%3E%3C/svg%3E)
Billy Cheng
0
Reputation points
commented 2024-05-20T05:20:52.2666667+00:00
PRADEEPCHEEKATLA-MSFT
78,986
Reputation points Microsoft Employee
1 answer
Error accessing Azure sql from Azure databricks using jdbc authentication=ActiveDirectoryInteractive
Getting below error while accessing Azure sql using jdbc from Azure databricks notebook, com.microsoft.sqlserver.jdbc.SQLServerException: Failed to authenticate the user p***** in Active Directory (Authentication=ActiveDirectoryInteractive). Unable to…
asked 2024-05-14T11:35:08.7566667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 9%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPM%3C/text%3E%3C/svg%3E)
Pankaj Mendi
0
Reputation points
commented 2024-05-20T05:10:21.33+00:00
PRADEEPCHEEKATLA-MSFT
78,986
Reputation points Microsoft Employee
0 answers
Is dynamic SQL Queries supported on Azur Databricks SQL Cluster?
Hello, I'm planning to implement Dynamic SQL function to query data on Databricks table. Tables and access for the users are governed by a custom access matrix using the Unity catalog. The problem is that in a custom matrix, there are two types of users:…
asked 2024-05-15T19:01:11.4766667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(64, 0%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJP%3C/text%3E%3C/svg%3E)
Jayesh Potwade
0
Reputation points
commented 2024-05-20T05:09:37.88+00:00
PRADEEPCHEEKATLA-MSFT
78,986
Reputation points Microsoft Employee
1 answer
[Databricks] Clusters are failing to launch. Cluster launch will be retried.
Hi all, I am a complete newbie on Databricks Azure. I have encounterd the below issue which I think is stopping me from running query. Any help will be much appreciated. Thanks. Billy Clusters are failing to launch. Cluster launch will be…
asked 2024-05-08T22:05:05.41+00:00
Billy Cheng
0
Reputation points
commented 2024-05-20T04:59:59.1233333+00:00
PRADEEPCHEEKATLA-MSFT
78,986
Reputation points Microsoft Employee
1 answer
Unable to downgrade Databricks workspace
When downgrading the Databricks Workspace, I receive the following message; However, none of the Enhanced Security options are currently enabled; Could you please help me identify the cause of the error?
asked 2024-05-16T09:02:23.1433333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(252.8, 73%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENM%3C/text%3E%3C/svg%3E)
Naomi Mostert | Codex
0
Reputation points
answered 2024-05-20T04:34:35.11+00:00
PRADEEPCHEEKATLA-MSFT
78,986
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Azure Databricks Lakehouse Monitoring queries
Hi Team, I was exploring on Azure Databricks Lakehouse monitoring. I have few queries on this: When I am running a "refresh metrics" irrespective of an automated schedule or manual refresh, which compute does it run? There is no mechanism to…
asked 2024-04-28T11:50:10.0333333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 35%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESG%3C/text%3E%3C/svg%3E)
Sudipta Goswami
20
Reputation points
commented 2024-05-18T11:31:49.4066667+00:00
Sudipta Goswami
20
Reputation points
0 answers
How to parse nested json array of document in ADF data flow
Hi all I am trying to fitch the values from a nested josh array of document , I have used aggregate to convert into objects but not able to fitch the values of all child nodes like as below itOffer.item itOffer.item.SplOfr itOffer.item.buy …
asked 2024-05-02T15:44:57.8633333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 6%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVR%3C/text%3E%3C/svg%3E)
venkat rao
45
Reputation points
edited a comment 2024-05-17T11:54:17.8533333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 61%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EP%3C/text%3E%3C/svg%3E)
phemanth
6,550
Reputation points Microsoft Vendor