Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,916 questions
1 answer
ADF-REST API Query Parameter Error
Hi Team, I have the scenario to ingest data from Rest api which has pagingToken .here I'm using Query Parameters in Relative Url and passing Value of Next PagingToken from Pagination by using Pagination Rule .I'm Getting Following Error.
asked 2022-05-24T05:59:00.6+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(211.20000000000002, 20%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPK%3C/text%3E%3C/svg%3E)
Prapul Kumar Dongari
116
Reputation points
answered 2022-05-25T17:27:28.48+00:00
ShaikMaheer-MSFT
37,896
Reputation points Microsoft Employee
1 answer
csv to hive database load
Hi Expert, How to load data from csv to Hive database via notebook
asked 2022-05-16T15:24:51.397+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(140.8, 75%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESR%3C/text%3E%3C/svg%3E)
Shambhu Rai
1,406
Reputation points
commented 2022-05-24T15:55:54.073+00:00
Shambhu Rai
1,406
Reputation points
3 answers
One of the answers was accepted by the question author.
Azure Databricks vs Adf
HI, I have experience using ADF and SSIS. I am now trying to understand and implement Databricks at work. From what I can see ADF has a GUI which is a simple drag and drop which runs on preconfigured spark clusters called Integrated run times. Databricks…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(300.8, 23%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EC%3C/text%3E%3C/svg%3E)
0 answers
Compare 2 indexes
Hello Azure Team, I want to compare two index and find out the missing data of one index. Can you please help me here. What is the process of comparing two indexes with diff schema? Thanks, Tejswi
asked 2022-05-20T11:20:42.213+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(243.2, 24%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJT%3C/text%3E%3C/svg%3E)
Jadhav, Tejswi
21
Reputation points
commented 2022-05-23T11:58:10.757+00:00
ShaikMaheer-MSFT
37,896
Reputation points Microsoft Employee
1 answer
azure databricks
i want to access my file which is in azure container name as test1. when I am uploading it I am having error shaded.databricks.org.apache.hadoop.fs.azure.AzureException: hadoop_azure_shaded.com.microsoft.azure.storage.StorageException: Server failed…
asked 2022-05-20T10:05:54.233+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(185.6, 84%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMA%3C/text%3E%3C/svg%3E)
Muqaddas Abbas
1
Reputation point
answered 2022-05-23T11:54:49.457+00:00
ShaikMaheer-MSFT
37,896
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Data decryption and decompression
Hi friends I have data encrypted and compression I want to decrypt decrypt and load in to snowflake how can I do in adf and with other services please help me whole steps
2 answers
databricks cluster creation
hello, i'm unable to launch a databricks cluster. I have tried various cluster type, different regions and am getting below error.. I'm using pay as you go standard plan. Any help would be appreciated. Cluster terminated.Reason:Cloud provider launch…
asked 2022-04-27T16:18:42.04+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(16, 1%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENK%3C/text%3E%3C/svg%3E)
Naveen Kaukantla
1
Reputation point
answered 2022-05-21T07:42:53.22+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(32, 41%, 13%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
Arunbalaji
1
Reputation point
1 answer
Databricks widgets
Databricks notebooks l have widgets can we use python parameterized function to make it modular and the widgets should be there in that?
asked 2022-05-05T11:04:06.743+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(166.4, 15%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
Anshal
1,866
Reputation points
commented 2022-05-19T15:48:10.387+00:00
ShaikMaheer-MSFT
37,896
Reputation points Microsoft Employee
1 answer
error reading delta parquet with new field added
Hello, loading daily delta parquet files into day folders each day i.e. ... /year=2022/month=05/day=09 /year=2022/month=05/day=10 today I added one more column to the load and so in day=11 the new field should be present This is what I use to…
asked 2022-05-11T06:24:10.03+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(262.40000000000003, 97%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
arkiboys
9,616
Reputation points
commented 2022-05-19T15:39:48.097+00:00
ShaikMaheer-MSFT
37,896
Reputation points Microsoft Employee
1 answer
Pass Filter Output to Data Bricks Notebook
How to pass filter activity output value to data bricks note book
asked 2022-05-11T09:58:21.757+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(275.2, 84%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKL%3C/text%3E%3C/svg%3E)
Krushnakanth Lenka
1
Reputation point
commented 2022-05-19T15:34:21.883+00:00
ShaikMaheer-MSFT
37,896
Reputation points Microsoft Employee
2 answers
DBrick workspace URL need toblock outside the corporate network?
As per security concern, need to restrict/block the dbricks workspace url outside the corporate network. Tried below ip access list, it able to restrict only user login access out the corporate network but still the workspace id url is live outside the…
asked 2022-05-12T21:25:02.84+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(179.20000000000002, 85%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
a1990
11
Reputation points
answered 2022-05-19T15:25:58.167+00:00
ShaikMaheer-MSFT
37,896
Reputation points Microsoft Employee
1 answer
Parquet column cannot be converted.
In ADF dataflow using the derivedcolumn, I convert the columns to the appropriate datatypes. i.e. string to date or to decimal... the sink is in delta parquet. Then in databricks I try to read the delta parquet but there is an error: Parquet…
0 answers
Enable dbutils.secrets.get with Databricks-Connect AzureDatabricks
I'm trying to use dbutils.secrets.get with databricks connect and the documentation says "Contact Azure Databricks support to enable this feature for your workspace" …
asked 2022-05-17T21:29:28.4+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(208, 72%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EWH%3C/text%3E%3C/svg%3E)
William Holbrook
1
Reputation point
commented 2022-05-18T09:10:48.497+00:00
PRADEEPCHEEKATLA-MSFT
76,836
Reputation points Microsoft Employee
0 answers
nltk and wheel files
All, I need to provide the ability for the users to use NLTK and several of its modules in Azure Databricks. I downloaded the wheel files of some of the modules but I'm unable to locate the wheel file for regex for Azure Databricks platform. The linux…
asked 2022-04-07T13:33:47.877+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(0, 24%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGR%3C/text%3E%3C/svg%3E)
Gopinath Rajee
646
Reputation points
commented 2022-05-17T21:54:22.797+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 9%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKM%3C/text%3E%3C/svg%3E)
KranthiPakala-MSFT
46,422
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Azure Databricks and Multi-Task Jobs and Notebook activity
All, We have lot of tables to load for which we plan to use notebook activity. We will read the set of tables call the notebook in a loop each time passing the next table in the loop. While the notebook activity method gives you the looping method it…
asked 2022-05-09T04:36:09.4+00:00
Gopinath Rajee
646
Reputation points
accepted 2022-05-17T05:20:18.233+00:00
Gopinath Rajee
646
Reputation points
1 answer
Databricks job timeout sending message to service bus with spark 3.2.1 and azure-messaging-servicebus sdk
Hi, we have developed a scala job using azure-messaging-servicebus sdk for java (ver 7.8.0) to send messages on service bus topic. All work fine as expected using an azure cloud databricks cluster 9.1 LTS (with Spark 3.1.2 and scala 2.12), but running…
asked 2022-05-13T10:32:19.103+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(64, 42%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMG%3C/text%3E%3C/svg%3E)
Martano, Giovanni
1
Reputation point
answered 2022-05-16T22:21:40.577+00:00
KranthiPakala-MSFT
46,422
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Azure Databricks - Running Jobs in Ineractive Cluster vs JobsCluster
All, Other than cost being one of the factors, Why would one want to use an Interactive Cluster (High Concurrency) to run nightly jobs instead of running them using JobClusters via Pools. Thanks, grajee
asked 2022-05-12T02:57:43.013+00:00
Gopinath Rajee
646
Reputation points
accepted 2022-05-16T14:41:51.127+00:00
Gopinath Rajee
646
Reputation points
1 answer
One of the answers was accepted by the question author.
csv to csv data load in adf
Hi Expert, how to load data from csv o csv using ADF nusing copy transformation or any readymade component
asked 2022-05-10T20:44:43.223+00:00
Shambhu Rai
1,406
Reputation points
accepted 2022-05-16T08:19:49.197+00:00
Shambhu Rai
1,406
Reputation points
1 answer
The selected table is not delta table.
Hi Expert, I am connecting Hive database using Azure Delta databricks datasets in azure data factory. connection and everything is successful but why i added in Sink and validate in copy data object .. it is giving validation error …
asked 2022-04-28T18:57:26.35+00:00
Shambhu Rai
1,406
Reputation points
commented 2022-05-16T04:46:07.153+00:00
Pratik Somaiya
4,201
Reputation points
2 answers
pyspark convert scientific notation to string
Something what should be really simple getting me frustrated. When reading from csv in pyspark in databricks the output has a scientific notation: Name Code AA 6.44E+11 BB 5.41E+12 how to convert it to string? Here is the…
asked 2021-09-23T22:19:51.847+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(60.8, 26%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EB%3C/text%3E%3C/svg%3E)
braxx
426
Reputation points
answered 2022-05-13T12:18:18.003+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(230.39999999999998, 94%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAK%3C/text%3E%3C/svg%3E)
Abhishek Kumar E
1
Reputation point