using streaming batch for multiple operations
I am new to spark and DataBricks and was trying to look for a solution where I can utilize a batch from a eventhub stream to accomplish multiple business logic but could not find any guidance. Stream I get from EventHub is a CDC stream from multiple…
![](https://techprofile.blob.core.windows.net/images/VfQFAmOikEWfBHko2XlWTA.png?8D7F33)
Databricks readstream writestream to Azure Synapse
I am having an issue on writing stream to Azure synapse with the following error . let's have a look and see if there is idea ?
![](https://techprofile.blob.core.windows.net/images/VfQFAmOikEWfBHko2XlWTA.png?8D7F33)
How to set security permissions to Databases in databricks through Notebooks
We are stuck on the way to set security permissions to Databases by using Notebooks %sql. At first, let me explain our situations and settings. We run the following code on Notebooks: %sql CREATE DATABASE X ; GRANT USAGE ON DATABASE X TO…
Databricks Pyspark exception handling best practices
Hi, In the current development of pyspark notebooks on Databricks, I typically use the python specific exception blocks to handle different situations that may arise. I am wondering if there are any best practices/recommendations or patterns to handle…
Azure Databricks Cluster
Hi, Created new cluster in Databricks (QA environment) Afte that when i try to click on data tab inroder to create database getting below error checked that cluster is up and running.
![](https://techprofile.blob.core.windows.net/images/VfQFAmOikEWfBHko2XlWTA.png?8D7F33)
Quota limit hit on tutorial notebook
I'm attempting to launch a default cluster (min 2, max 8) on the premium trial account in order to run 01-The-Databricks-Environment. I haven't been able to run any operations in the notebook. Receiving this error: Azure error code:…
![](https://techprofile.blob.core.windows.net/images/VfQFAmOikEWfBHko2XlWTA.png?8D7F33)
Streaming upserts constantly report mysterious log, are very slow
Our streaming batch upserts are very slow and constantly report: 20/12/03 23:38:04 WARN RetryTolerableRenameFSDataOutputStream: Failed to rename temp file…
M5P from PackageRWeka
After running that command for R (from Machine Learning with R) in Azure Databricks Notebook: library(SparkR) library(RWeka) m.m5p <- M5P(quality ~ alcohol + sulphates , data = wine_train) I obtained this : Error in .jcall(o,…
![](https://techprofile.blob.core.windows.net/images/VfQFAmOikEWfBHko2XlWTA.png?8D7F33)
Databricks 7.3ML vs 7.0ML
Hi, I'm trying to run a distributed training job using Horovod and Pytorch on Azure Databricks using 2 nodes with 2 gpus each. When I run my code on version 7.0ML, the code sees each node as having 2 GPUs which is expected behavior. However, on 7.3ML,…
![](https://techprofile.blob.core.windows.net/images/VfQFAmOikEWfBHko2XlWTA.png?8D7F33)
Could you please let me know how reliable to use OpenCensus library in azure databricks for logging and any Microsoft support available in case of product issues?
We would like to use Opencensus in Azure Data Bricks with Python for logging with Azure Monitor. Could you please let me know how reliable this library to use in our application and any Microsoft support available in case of any product issues?
![](https://techprofile.blob.core.windows.net/images/VfQFAmOikEWfBHko2XlWTA.png?8D7F33)
Service principal for custom Spark app
I've got a custom Spark app built on top of a custom PaaS (OpenShift based) deployed on Azure IaaS. Is it possible to assign a service principal to my app for the purpose of access control? If so, how do I do it?
Register Azure ML Model from DatabricksStep
Hi, I'm calculating a model while executing a DatabricksStep in an Azure ML Pipeline, save it on my Blob Storage as .pkl file and upload it to the current Azure ML Run using Run.upload_file (). All this works without any problems. But as soon as I try to…
Azure Databricks - How to restrict users don't spinup clusters with "L" Series?
How to create or change cluster policy (Spark) in Databricks to restrict users don't spinup clusters with "L" Series? to restrict users to change from existing series to "L"?
data factory - databricks
Hello, I am in the process of learning the data factory. 1- Is there anything that data factory/data flow does not do which means it needs to be done in databricks? I am trying to find-out that if I know data factory really well, then is there a…
¿Tengo el servicio si soy de CCH?
Al iniciar la sesión y querer obtener el servicio no me deja, me marca un error. AADSTS50011: The reply URL specified in the request does not match the reply URLs configured for the application: '8e0e8db5-b713-4e91-98e6-470fed0aa4c2'. Spanish =>…
![](https://techprofile.blob.core.windows.net/images/VfQFAmOikEWfBHko2XlWTA.png?8D7F33)
You do not have access
You do not have access Sorry but our systems are a detecting a problem with your account and are unable to validate your access. Please contact Support and reference the following transaction ID and message code. TransactionId:…
![](https://techprofile.blob.core.windows.net/images/VfQFAmOikEWfBHko2XlWTA.png?8D7F33)
Learning databricks
Hello, I have been searching online websites and youtube for good step by step learning tutorials which I can follow along to learn databricks as well as databricks inside ADF. 1- Can you please recommend websites or youtube channels? 2- I do…
Handshake fails trying to connect from Azure Databricks to Azure PostgreSQL with SSL
We're trying to connect to an Azure Database for PostgreSQL flexible server from Databricks using the jdbc driver org.postgresql.Driver. Since the flexible server enforces SSL, we added the ssl and sslmode options to our existing code: driver =…
![](https://techprofile.blob.core.windows.net/images/VfQFAmOikEWfBHko2XlWTA.png?8D7F33)
Convert dataframe column datatypes according to target table column datatypes - Databricks
Hi All, I am currently loading the csv file, all with string datatypes columns to dataframe df. I want to write this data into an existing table which has different datatypes. So basically in csv file I have a column CreatedDate as string and my…
![](https://techprofile.blob.core.windows.net/images/VfQFAmOikEWfBHko2XlWTA.png?8D7F33)
Custom Managed key for Azure Databricks Notebook
How to set customr managed key for notebook in Azure daabricks