Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,399 questions
1 answer
Where to store delta-files
Hey guys! I have been wondering about the below question for a while and I hope you can help me get a good night sleep again. Question: When you are working in Azure DataBricks to do transformations, you can save the results using a delta format.…
asked 2020-10-05T13:05:54.297+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(99.2, 21%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECG%3C/text%3E%3C/svg%3E)
Christian Gert Hansen
1
Reputation point
commented 2020-10-08T20:15:56.767+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 26%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HimanshuSinha-msft
19,386
Reputation points Microsoft Employee
1 answer
DatabricksL Error Code 3208
This issue was already reported in Git50030Hub. We have the same issue. Very little technical details here, can someone from Microsoft add more technical details? Microsoft's t-shooting doc here is very amusing. This issued showed up in our newly created…
asked 2020-10-06T19:08:16.867+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 33%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPM%3C/text%3E%3C/svg%3E)
Porsche Me
131
Reputation points
commented 2020-10-08T10:23:40.55+00:00
PRADEEPCHEEKATLA-MSFT
83,301
Reputation points Microsoft Employee
1 answer
'databricks configure --token' command waiting for the input
I run following task in Azure DevOps, it always waiting for input? Why wasn't my bash automatic supply working? Python 3.7 installed databricksUrl=https://... databricksToken=***** databricks configure --token…
asked 2020-10-01T04:37:38.963+00:00
Porsche Me
131
Reputation points
commented 2020-10-07T13:05:45.757+00:00
PRADEEPCHEEKATLA-MSFT
83,301
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Opening up port 443 on Azure Databricks (managed resource)
Hello, I was wondering if there is a way to open port 443 (HTTPS) on a managed azure databricks workspace. . I've attempted to attach a network interface and network security group components on the same resource group and added inbound rule 443 but…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(76.8, 25%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPK%3C/text%3E%3C/svg%3E)
2 answers
AD Authentication to Databricks REST API Returns 403 Unauthorized
I'm attempting to call the Databricks API using the method described in Get an Azure Active Directory token using a service principal. However when I use the management endpoint access token to access the Databricks REST API, I get HTTP ERROR 403.…
asked 2020-06-29T16:03:39.38+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(227.2, 81%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EML%3C/text%3E%3C/svg%3E)
M.Lo
1
Reputation point
commented 2020-10-02T08:27:10.42+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(128, 59%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPJ%3C/text%3E%3C/svg%3E)
Paweł Jankowski
1
Reputation point
2 answers
Databricks cluster does not work with free trial subscription
In Azure Databricks, tried to create cluster. Cluster creation requires to provide worker type and driver type, in which minimum 4 cores are required on each which means in total 8 cores. Free trail subscription gives only 4 cores what I read. So how…
asked 2020-06-11T22:24:30.773+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(73.60000000000001, 42%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESD%3C/text%3E%3C/svg%3E)
Seuli Das Majumder
101
Reputation points
answered 2020-10-01T14:36:15.337+00:00
PRADEEPCHEEKATLA-MSFT
83,301
Reputation points Microsoft Employee
1 answer
dbutils.widgets.dropdown() issue: com.databricks.dbutils_v1.TooManyDefaultChoices: Too many default choices (1116). Limit is 1024
HI, While running dbutils.widgets.dropdown(), I face the following issue: com.databricks.dbutils_v1.TooManyDefaultChoices: Too many default choices (1116). Limit is 1024. I searched but could not find an answer, how to change the default value. For sure,…
asked 2020-09-21T12:44:16.483+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(233.6, 37%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECK%3C/text%3E%3C/svg%3E)
Chaitanya Kumar Dondapati (Volkswagen)
1
Reputation point
commented 2020-09-30T19:48:14.75+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(172.8, 70%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
MartinJaffer-MSFT
26,046
Reputation points
1 answer
Mounting entire ADLS on azure databricks
Hi, I want to mount entire ADLS storage account on databricks. I've checked in documents I can mount a single filesystem at a time but I want to mount entire ADLS on databricks. I've around 70 containers in my ADLS and I want to mount all of them…
asked 2020-09-23T11:35:43.993+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(76.8, 38%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAG%3C/text%3E%3C/svg%3E)
Ashima Gupta
1
Reputation point
commented 2020-09-29T19:04:09.19+00:00
MartinJaffer-MSFT
26,046
Reputation points
1 answer
One of the answers was accepted by the question author.
remainingCores is too small error. Spark and Kafka cluster on HDInsight problem
I tried to create a Kafka and Spark cluster in the HDInsight using either of templates in these links: …
asked 2020-09-27T06:57:32.923+00:00
Erfan Mahmoodnejad
41
Reputation points
commented 2020-09-29T06:31:58.623+00:00
PRADEEPCHEEKATLA-MSFT
83,301
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Can i access Gen1 and Gen 2 datalake sources from the same databricks cluster
If I have access to Gen 1 datalake sources on databricks cluster. Can I add Gen2 sources on the same databricks cluster and access them both Gen 1 and Gen 2 from the same databricks cluster Or Do i have to create separate cluster for connecting to gen…
asked 2020-09-27T05:54:11.377+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(137.6, 0%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAG%3C/text%3E%3C/svg%3E)
Abhishek Gaikwad
191
Reputation points
accepted 2020-09-29T03:56:29.637+00:00
Abhishek Gaikwad
191
Reputation points
1 answer
Calling Azure Databricks query from Azure Function
Hi Everyone, Is possible to call a SQL query in Azure Databricks(Cluster) inside a Azure Function ? Normally to conenct SQL server from Azure function, we use to create connection string in application seetings and use to retrive using environmnet…
asked 2020-09-16T13:24:48.947+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(300.8, 84%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EG%3C/text%3E%3C/svg%3E)
Gowrikanth
1
Reputation point
commented 2020-09-25T08:50:42.997+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(51.2, 79%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKG%3C/text%3E%3C/svg%3E)
Krish G
2,331
Reputation points
0 answers
Is it possible to add a Databricks Role/Group to appRoles for OAuth 2.0(Azure AD) tokens
I have configured a service principal for Azure Databricks in Azure AD as per this documentation: https://learn.microsoft.com/en-us/azure/databricks/dev-tools/api/latest/aad/service-prin-aad-token I am able to access Azure Databricks using an OAuth…
asked 2020-09-09T11:48:28.817+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(233.6, 4%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMR%3C/text%3E%3C/svg%3E)
Mukil Rajeev
1
Reputation point
commented 2020-09-16T22:00:52.237+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 9%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKM%3C/text%3E%3C/svg%3E)
KranthiPakala-MSFT
46,432
Reputation points Microsoft Employee
1 answer
Alternative to using Azure SQL Managed Instance for Complex calculations
I have inherited an Architecture which carries out complex calculation in Azure SQL Managed Instance via Databricks. The Databricks is connected via Apache Spark connector. The Databricks does the calculations in SQL and the end results are in Tables…
asked 2020-09-02T16:31:09.39+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 90%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EK%3C/text%3E%3C/svg%3E)
Kman
41
Reputation points
commented 2020-09-16T19:45:52.69+00:00
MartinJaffer-MSFT
26,046
Reputation points
1 answer
Databricks certification in Azure
Hello team, I am planning to do certification in Databricks.So I just wanted to know is there any certificate on databricks on Azure if yes please advise the exam number otherwise which one i can get certified. Please advise Thanks in advance …
asked 2020-09-16T17:01:16.283+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(211.20000000000002, 24%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKR%3C/text%3E%3C/svg%3E)
Kulkarni, Rohit
1
Reputation point
answered 2020-09-16T17:02:51.113+00:00
Leon Laude
85,701
Reputation points
1 answer
Not able to delete any resource
Please help me regarding this error. Not able to delete resource group.
asked 2020-09-10T09:57:36.12+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(278.4, 89%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDC%3C/text%3E%3C/svg%3E)
demo case
1
Reputation point
commented 2020-09-14T07:51:40.453+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(108.80000000000001, 9%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HarithaMaddi-MSFT
10,136
Reputation points
1 answer
Apache Spark Event Hubs connector OAuth authentication - is it supported?
Does anyone has an example of reading streams from Azure EventHub using OAuth authentication in Databricks? I find only examples which use SAS keys and to be honest I am sure whether this is officially supported.
asked 2020-09-08T07:58:49.787+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(316.8, 67%, 40%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGS%3C/text%3E%3C/svg%3E)
Georgi Slavov
1
Reputation point
answered 2020-09-10T09:29:59.22+00:00
PRADEEPCHEEKATLA-MSFT
83,301
Reputation points Microsoft Employee
1 answer
Security Recommendations for Azure Data and Analytics Services
I am working on Securing Data and Analytics Services on Azure. I want to know what security controls i can apply after creating of services and what i can apply only during the service creation. Below are the recommendation i have found as of now. Could…
asked 2020-09-02T20:20:43.98+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(137.6, 61%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAV%3C/text%3E%3C/svg%3E)
Akash Verma
21
Reputation points
commented 2020-09-09T11:11:32.767+00:00
PRADEEPCHEEKATLA-MSFT
83,301
Reputation points Microsoft Employee
1 answer
how add class library for BeautifulSoup in Azure Databricks
hi all, how add class library for BeautifulSoup in Azure Data-bricks i want to run below code in pyspark notebook from bs4 import BeautifulSoup import pandas as pd table = BeautifulSoup(open('C:/age0.html','r').read()).find('table') df =…
asked 2020-08-27T07:00:54.517+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(153.6, 57.00000000000001%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVM%3C/text%3E%3C/svg%3E)
Verma, Manish Kumar
131
Reputation points
commented 2020-09-02T04:36:38.853+00:00
PRADEEPCHEEKATLA-MSFT
83,301
Reputation points Microsoft Employee
1 answer
Want to Read Microsoft Team Post in Azure Data Factory
I Have requirement to read Microsoft Team post in Azure Data Factory. currently i am using logic apps , but it generate output in html file and it is difficult to read what is the best way to get data from Microsoft team, so we can read data in…
asked 2020-08-27T08:52:24.843+00:00
Verma, Manish Kumar
131
Reputation points
commented 2020-08-31T06:13:56.377+00:00
MayankBargali-MSFT
69,931
Reputation points
2 answers
Is there any other free streaming sources like twitter for practicing spark and Kafka
Friends I am learning Kafka and Spark. I worked in Kafka and spark integration using Twitter api but I want to do more practice Is there any other free streaming sources like twitter for practicing spark and Kafka
asked 2020-08-25T18:56:42.463+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(9.6, 76%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESB%3C/text%3E%3C/svg%3E)
Surendiran Balasubramanian
1
Reputation point
commented 2020-08-31T04:31:30.637+00:00
PRADEEPCHEEKATLA-MSFT
83,301
Reputation points Microsoft Employee