Unable to Create Azure Databricks SQL Endpoint
Hi, I am unable to create an Azure Databricks SQL Endpoint. I confirm that I created the Databricks cluster using the same admin account that I am now attempting to create the Databricks SQL Endpoint. So when launching Databricks I am…
Azure Copy data from one SQL pool to another SQL pool
Hi, I have a scenario where I need to create a pipeline that copies data from one SQL pool to another SQL pool. Can you please suggest how that is done. Thanks in advance.
How to install azure cli on databricks
Hi Team, I want to install azure cli in databricks nodes ..could u pls help on how to install it ..
ADF-REST API Query Parameter Error
Hi Team, I have the scenario to ingest data from Rest api which has pagingToken .here I'm using Query Parameters in Relative Url and passing Value of Next PagingToken from Pagination by using Pagination Rule .I'm Getting Following Error.
csv to hive database load
Hi Expert, How to load data from csv to Hive database via notebook
Azure Databricks vs Adf
HI, I have experience using ADF and SSIS. I am now trying to understand and implement Databricks at work. From what I can see ADF has a GUI which is a simple drag and drop which runs on preconfigured spark clusters called Integrated run times. Databricks…
Compare 2 indexes
Hello Azure Team, I want to compare two index and find out the missing data of one index. Can you please help me here. What is the process of comparing two indexes with diff schema? Thanks, Tejswi
azure databricks
i want to access my file which is in azure container name as test1. when I am uploading it I am having error shaded.databricks.org.apache.hadoop.fs.azure.AzureException: hadoop_azure_shaded.com.microsoft.azure.storage.StorageException: Server failed…
Data decryption and decompression
Hi friends I have data encrypted and compression I want to decrypt decrypt and load in to snowflake how can I do in adf and with other services please help me whole steps
databricks cluster creation
hello, i'm unable to launch a databricks cluster. I have tried various cluster type, different regions and am getting below error.. I'm using pay as you go standard plan. Any help would be appreciated. Cluster terminated.Reason:Cloud provider launch…
Databricks widgets
Databricks notebooks l have widgets can we use python parameterized function to make it modular and the widgets should be there in that?
error reading delta parquet with new field added
Hello, loading daily delta parquet files into day folders each day i.e. ... /year=2022/month=05/day=09 /year=2022/month=05/day=10 today I added one more column to the load and so in day=11 the new field should be present This is what I use to…
Pass Filter Output to Data Bricks Notebook
How to pass filter activity output value to data bricks note book
DBrick workspace URL need toblock outside the corporate network?
As per security concern, need to restrict/block the dbricks workspace url outside the corporate network. Tried below ip access list, it able to restrict only user login access out the corporate network but still the workspace id url is live outside the…
Parquet column cannot be converted.
In ADF dataflow using the derivedcolumn, I convert the columns to the appropriate datatypes. i.e. string to date or to decimal... the sink is in delta parquet. Then in databricks I try to read the delta parquet but there is an error: Parquet…
Enable dbutils.secrets.get with Databricks-Connect AzureDatabricks
I'm trying to use dbutils.secrets.get with databricks connect and the documentation says "Contact Azure Databricks support to enable this feature for your workspace" …
nltk and wheel files
All, I need to provide the ability for the users to use NLTK and several of its modules in Azure Databricks. I downloaded the wheel files of some of the modules but I'm unable to locate the wheel file for regex for Azure Databricks platform. The linux…
Azure Databricks and Multi-Task Jobs and Notebook activity
All, We have lot of tables to load for which we plan to use notebook activity. We will read the set of tables call the notebook in a loop each time passing the next table in the loop. While the notebook activity method gives you the looping method it…
Databricks job timeout sending message to service bus with spark 3.2.1 and azure-messaging-servicebus sdk
Hi, we have developed a scala job using azure-messaging-servicebus sdk for java (ver 7.8.0) to send messages on service bus topic. All work fine as expected using an azure cloud databricks cluster 9.1 LTS (with Spark 3.1.2 and scala 2.12), but running…
Azure Databricks - Running Jobs in Ineractive Cluster vs JobsCluster
All, Other than cost being one of the factors, Why would one want to use an Interactive Cluster (High Concurrency) to run nightly jobs instead of running them using JobClusters via Pools. Thanks, grajee