Azure Databricks

How can I resolve this error marketplace ??

How to call websocket api and make a data set based on time?

I want to call websocket api from data anlytics or data factory which gives continous data from api. i want to read that data and based on time after every 5 minutes want to take out open high low and close values from that set. i can able to do it using…

Azure databricks workload type

I am calling a azure databricks cluster from a AI/ML pipeline. I want to understand it will get executed under which feature type: Data analytics or data engineering. I want to understand this to get my databricks usage cost calculation.

Why we dont have diagnostic settings feature in Monitoring Section of "Azure Databricks of standard version"

Hi, I want to monitor Azure Databricks using Azure monitor .. I first want to send all its diagnostics settings to log analytics and then i can analyze from there. but there is no diagnostics setting feature Available for "Azure databricks Standard…

Azure Databricks table

Hi, One of my Datascientist by mistakenly delete data file working on Azure Databricks. Now How to restore deleted table from Azure Data Lake?

How to maintain uniqueness in dataframe in databricks

When you load data into dataframes in databricks how can you make sure the rows in dataframes are not duplicated. In SQL you can handle by using unique constraint on the tables. how this can be handle in dataframes to ensure rows are not duplicated.

Getting SSL certificate error while connecting to Synapse SQL pool from Databricks in Azure using JDBC

--------------------------------------------------------------------------- Py4JJavaError Traceback (most recent call last) <command-3031308706598466> in <module> 6 .option("tempDir",…

notebook-scoped lib failing to find the dependent cluster-scoped lib

Our app hit an issue with ModuleNotFoundError recently with Databricks Runtime 6.4 and it looks this is a databricks issue, so wanted to ask about that. The problem can happen as follows: Two libraries A.B and A.C; A.C depends on A.B Install A.B…

Unable to read csv file from cosmos gen1 through spark. Pandas read works.

This works, df = pd.read_csv('/dbfs/mnt/ajviswan/forest_efficiency/2020-04-26_2020-05-26.csv') sdf = spark.createDataFrame(df) sdf.head() But spark read does not work. df =…

Azure Databricks Access Controls in Programmatic way

Can someone please let me know is there any programmatic way to implement Access control on Azure Databricks on Workspace object, cluster, pool, job, and table etc.. Any powershell or API for the same ?

Azure Databrick workspace linking to Azure Machine Learning workspace error unable to get Workspace.from_config()

We are in a POC of Azure Databricks and Azure Machine Learning integration so that we can advance our MLOps practice. We have been dealing the with workspace linking issue for the past few weeks and have hit a wall. We worked with Databricks and search…

Writing parquet file throws…An HTTP header that's mandatory for this request is not specified

I have two ADLSv2 storage accounts, both are hierarchical namespace enabled. In my Python Notebook, I'm reading a CSV file from one storage account and writing as parquet file in another storage, after some enrichment. I am getting below error when…

is there any limitation of writing billions of record from Azure Data bricks to Azure SQL using JDBC

Hi All, we are follow up this url- https://learn.microsoft.com/en-us/azure/databricks/data/data-sources/sql-databases we want to know is there any limitation of writing billions of record from Azure Data bricks to Azure SQL using JDBC …

Where to store delta-files

Hey guys! I have been wondering about the below question for a while and I hope you can help me get a good night sleep again. Question: When you are working in Azure DataBricks to do transformations, you can save the results using a delta format.…

DatabricksL Error Code 3208

This issue was already reported in Git50030Hub. We have the same issue. Very little technical details here, can someone from Microsoft add more technical details? Microsoft's t-shooting doc here is very amusing. This issued showed up in our newly created…

'databricks configure --token' command waiting for the input

I run following task in Azure DevOps, it always waiting for input? Why wasn't my bash automatic supply working? Python 3.7 installed databricksUrl=https://... databricksToken=***** databricks configure --token…

Opening up port 443 on Azure Databricks (managed resource)

Hello, I was wondering if there is a way to open port 443 (HTTPS) on a managed azure databricks workspace. . I've attempted to attach a network interface and network security group components on the same resource group and added inbound rule 443 but…

AD Authentication to Databricks REST API Returns 403 Unauthorized

I'm attempting to call the Databricks API using the method described in Get an Azure Active Directory token using a service principal. However when I use the management endpoint access token to access the Databricks REST API, I get HTTP ERROR 403.…

Databricks cluster does not work with free trial subscription

In Azure Databricks, tried to create cluster. Cluster creation requires to provide worker type and driver type, in which minimum 4 cores are required on each which means in total 8 cores. Free trail subscription gives only 4 cores what I read. So how…

dbutils.widgets.dropdown() issue: com.databricks.dbutils_v1.TooManyDefaultChoices: Too many default choices (1116). Limit is 1024

HI, While running dbutils.widgets.dropdown(), I face the following issue: com.databricks.dbutils_v1.TooManyDefaultChoices: Too many default choices (1116). Limit is 1024. I searched but could not find an answer, how to change the default value. For sure,…