Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,116 questions
1 answer
One of the answers was accepted by the question author.
using streaming batch for multiple operations
I am new to spark and DataBricks and was trying to look for a solution where I can utilize a batch from a eventhub stream to accomplish multiple business logic but could not find any guidance. Stream I get from EventHub is a CDC stream from multiple…
asked 2020-12-15T04:00:40.087+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(54.400000000000006, 2%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERS%3C/text%3E%3C/svg%3E)
Rohit Sapru
41
Reputation points
commented 2020-12-16T05:15:45.41+00:00
PRADEEPCHEEKATLA-MSFT
87,326
Reputation points Microsoft Employee
1 answer
Databricks readstream writestream to Azure Synapse
I am having an issue on writing stream to Azure synapse with the following error . let's have a look and see if there is idea ?
asked 2020-12-10T06:08:39.147+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(137.6, 40%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
sakuraime
2,321
Reputation points
commented 2020-12-15T06:19:23.62+00:00
PRADEEPCHEEKATLA-MSFT
87,326
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
How to set security permissions to Databases in databricks through Notebooks
We are stuck on the way to set security permissions to Databases by using Notebooks %sql. At first, let me explain our situations and settings. We run the following code on Notebooks: %sql CREATE DATABASE X ; GRANT USAGE ON DATABASE X TO…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(233.6, 8%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
1 answer
One of the answers was accepted by the question author.
Databricks Pyspark exception handling best practices
Hi, In the current development of pyspark notebooks on Databricks, I typically use the python specific exception blocks to handle different situations that may arise. I am wondering if there are any best practices/recommendations or patterns to handle…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(51.2, 87%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESD%3C/text%3E%3C/svg%3E)
1 answer
Azure Databricks Cluster
Hi, Created new cluster in Databricks (QA environment) Afte that when i try to click on data tab inroder to create database getting below error checked that cluster is up and running.
asked 2020-11-25T07:27:39.867+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(16, 46%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVK%3C/text%3E%3C/svg%3E)
Vijay Kumar
2,031
Reputation points
commented 2020-12-11T09:29:48.567+00:00
PRADEEPCHEEKATLA-MSFT
87,326
Reputation points Microsoft Employee
2 answers
Quota limit hit on tutorial notebook
I'm attempting to launch a default cluster (min 2, max 8) on the premium trial account in order to run 01-The-Databricks-Environment. I haven't been able to run any operations in the notebook. Receiving this error: Azure error code:…
asked 2020-12-07T20:09:56.14+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 50%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EZW%3C/text%3E%3C/svg%3E)
Zak Wear
1
Reputation point
commented 2020-12-11T04:53:19.223+00:00
PRADEEPCHEEKATLA-MSFT
87,326
Reputation points Microsoft Employee
0 answers
Streaming upserts constantly report mysterious log, are very slow
Our streaming batch upserts are very slow and constantly report: 20/12/03 23:38:04 WARN RetryTolerableRenameFSDataOutputStream: Failed to rename temp file…
asked 2020-12-03T23:40:38.943+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(169.60000000000002, 22%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAH%3C/text%3E%3C/svg%3E)
Alex Hoffer
1
Reputation point
commented 2020-12-10T20:48:08.277+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 26%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HimanshuSinha-msft
19,466
Reputation points Microsoft Employee
0 answers
M5P from PackageRWeka
After running that command for R (from Machine Learning with R) in Azure Databricks Notebook: library(SparkR) library(RWeka) m.m5p <- M5P(quality ~ alcohol + sulphates , data = wine_train) I obtained this : Error in .jcall(o,…
asked 2020-11-12T07:06:42.407+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(249.60000000000002, 55.00000000000001%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMB%3C/text%3E%3C/svg%3E)
Mohamed Bah
1
Reputation point
commented 2020-12-08T10:53:41.693+00:00
PRADEEPCHEEKATLA-MSFT
87,326
Reputation points Microsoft Employee
0 answers
Databricks 7.3ML vs 7.0ML
Hi, I'm trying to run a distributed training job using Horovod and Pytorch on Azure Databricks using 2 nodes with 2 gpus each. When I run my code on version 7.0ML, the code sees each node as having 2 GPUs which is expected behavior. However, on 7.3ML,…
asked 2020-11-22T22:47:23.307+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(3.2, 35%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERP%3C/text%3E%3C/svg%3E)
Rohit Pillai
1
Reputation point Microsoft Employee
commented 2020-12-08T10:52:23.833+00:00
PRADEEPCHEEKATLA-MSFT
87,326
Reputation points Microsoft Employee
1 answer
Could you please let me know how reliable to use OpenCensus library in azure databricks for logging and any Microsoft support available in case of product issues?
We would like to use Opencensus in Azure Data Bricks with Python for logging with Azure Monitor. Could you please let me know how reliable this library to use in our application and any Microsoft support available in case of any product issues?
asked 2020-11-25T18:06:26.49+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(70.4, 44%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ETN%3C/text%3E%3C/svg%3E)
Tanguturi, Naresh
1
Reputation point
answered 2020-12-03T06:34:39.97+00:00
PRADEEPCHEEKATLA-MSFT
87,326
Reputation points Microsoft Employee
2 answers
One of the answers was accepted by the question author.
Service principal for custom Spark app
I've got a custom Spark app built on top of a custom PaaS (OpenShift based) deployed on Azure IaaS. Is it possible to assign a service principal to my app for the purpose of access control? If so, how do I do it?
asked 2020-11-25T11:42:14.303+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(304, 75%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMN%3C/text%3E%3C/svg%3E)
Minh NGUYEN
126
Reputation points
accepted 2020-12-01T12:50:14.973+00:00
Minh NGUYEN
126
Reputation points
1 answer
Register Azure ML Model from DatabricksStep
Hi, I'm calculating a model while executing a DatabricksStep in an Azure ML Pipeline, save it on my Blob Storage as .pkl file and upload it to the current Azure ML Run using Run.upload_file (). All this works without any problems. But as soon as I try to…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(144, 78%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJ%3C/text%3E%3C/svg%3E)
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(22.400000000000002, 11%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERM%3C/text%3E%3C/svg%3E)
1 answer
One of the answers was accepted by the question author.
Azure Databricks - How to restrict users don't spinup clusters with "L" Series?
How to create or change cluster policy (Spark) in Databricks to restrict users don't spinup clusters with "L" Series? to restrict users to change from existing series to "L"?
asked 2020-11-23T21:15:49.287+00:00
Vijay Kumar
2,031
Reputation points
accepted 2020-11-30T17:30:06.383+00:00
Vijay Kumar
2,031
Reputation points
1 answer
One of the answers was accepted by the question author.
data factory - databricks
Hello, I am in the process of learning the data factory. 1- Is there anything that data factory/data flow does not do which means it needs to be done in databricks? I am trying to find-out that if I know data factory really well, then is there a…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(262.40000000000003, 97%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
0 answers
¿Tengo el servicio si soy de CCH?
Al iniciar la sesión y querer obtener el servicio no me deja, me marca un error. AADSTS50011: The reply URL specified in the request does not match the reply URLs configured for the application: '8e0e8db5-b713-4e91-98e6-470fed0aa4c2'. Spanish =>…
asked 2020-11-21T21:00:30.487+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(128, 8%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
Anonymous
commented 2020-11-26T12:18:34.817+00:00
PRADEEPCHEEKATLA-MSFT
87,326
Reputation points Microsoft Employee
0 answers
You do not have access
You do not have access Sorry but our systems are a detecting a problem with your account and are unable to validate your access. Please contact Support and reference the following transaction ID and message code. TransactionId:…
asked 2020-11-24T20:25:11.083+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(220.8, 87%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESR%3C/text%3E%3C/svg%3E)
Steve Rock
1
Reputation point
commented 2020-11-26T12:07:26.73+00:00
PRADEEPCHEEKATLA-MSFT
87,326
Reputation points Microsoft Employee
2 answers
One of the answers was accepted by the question author.
Learning databricks
Hello, I have been searching online websites and youtube for good step by step learning tutorials which I can follow along to learn databricks as well as databricks inside ADF. 1- Can you please recommend websites or youtube channels? 2- I do…
1 answer
One of the answers was accepted by the question author.
Handshake fails trying to connect from Azure Databricks to Azure PostgreSQL with SSL
We're trying to connect to an Azure Database for PostgreSQL flexible server from Databricks using the jdbc driver org.postgresql.Driver. Since the flexible server enforces SSL, we added the ssl and sslmode options to our existing code: driver =…
asked 2020-11-21T12:23:57.117+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(252.8, 71%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ETH%3C/text%3E%3C/svg%3E)
Tobias Himmighöfer
21
Reputation points
commented 2020-11-24T07:08:31.24+00:00
PRADEEPCHEEKATLA-MSFT
87,326
Reputation points Microsoft Employee
2 answers
One of the answers was accepted by the question author.
Convert dataframe column datatypes according to target table column datatypes - Databricks
Hi All, I am currently loading the csv file, all with string datatypes columns to dataframe df. I want to write this data into an existing table which has different datatypes. So basically in csv file I have a column CreatedDate as string and my…
asked 2020-11-17T11:39:40.01+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(275.2, 98%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAS%3C/text%3E%3C/svg%3E)
Ashish Sinha
161
Reputation points
commented 2020-11-20T04:57:55.493+00:00
PRADEEPCHEEKATLA-MSFT
87,326
Reputation points Microsoft Employee
1 answer
Custom Managed key for Azure Databricks Notebook
How to set customr managed key for notebook in Azure daabricks
asked 2020-11-04T10:35:37.78+00:00
Anonymous
commented 2020-11-19T17:54:07.127+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 44%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESS%3C/text%3E%3C/svg%3E)
Saurabh Sharma
23,796
Reputation points Microsoft Employee