Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,406 questions
connection instability between datafactory and databricks
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(291.2, 28.000000000000004%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ELP%3C/text%3E%3C/svg%3E)
Lionel POCKAM
20
Reputation points
Hello i'm facing a connectivity issue between DF agents on VM Windows and Databricks cluster. The agents on the windows sometimes facing network disconnection and the pipeline does not complete entirely.Anyone who have facing this issue can you give some advices ?
{count} votes
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 9%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKM%3C/text%3E%3C/svg%3E)
KranthiPakala-MSFT 46,632 Reputation points Microsoft Employee2023-07-31T23:15:14.5933333+00:00 @Lionel POCKAM Welcome to Microsoft Q&A forum and thanks for reaching out here.
It seems like the connection issue between ADF SHIR and Databricks cluster is intermittent.
Could you please confirm if you have got a chance to look at the SHIR logs from event viewer in SHIR Hosted VM to see what exactly has resulted in this intermittent issue? But the issue looks like a networking related and hence it may need deeper investigation of the logs to better understand the problem.To work around the problem, could you please try to implement retry mechanism so that even if there is any temporary network issue between SHIR and Databricks cluster, it will auto resolve in short time.
Kindly try the above workaround and let me know how it goes.
Thank you
-
Lionel POCKAM 20 Reputation points
2023-08-01T14:10:06.1533333+00:00 Hello @KranthiPakala-MSFT thank for your response. I have a called today with the ADF team and Databricks Core it seems that the IP access-list of Databricks cluster do not accept the connection with some public IP of ADF. We are awaiting of a change management process to add this ip on the Firewall and Databricks cluster.
-
KranthiPakala-MSFT 46,632 Reputation points Microsoft Employee
2023-08-01T16:16:52.75+00:00 @Lionel POCKAM Thanks for sharing additional information about the problem.
If you have already filed a support ticket with MSFT, could you please share the SR# with us here, so that we can track it internally.
Feel free to post the final resolution details once the issue has been resolved.
Thank you
-
Lionel POCKAM 20 Reputation points
2023-08-02T08:28:12.1033333+00:00 Hello @KranthiPakala-MSFT the ID ticket is ID#2307100050001283 effectively the solution proposed let me in trouble because things work before with private network and private endpoint.
-
KranthiPakala-MSFT 46,632 Reputation points Microsoft Employee
2023-08-02T18:19:33.66+00:00 @Lionel POCKAM Appreciate much for sharing the details! Will review the ticket and update back here. Thanks
Sign in to comment
Sign in to answer