Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
Spark cluster not found in ADF
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(140.8, 66%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPC%3C/text%3E%3C/svg%3E)
prathyusha chennareddy
6
Reputation points
When a dataflow is executed from pipeline, we got the below error.
Azure Data Factory
{count} vote
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(240, 12%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESA%3C/text%3E%3C/svg%3E)
Samy Abdul 3,376 Reputation points2023-01-06T13:07:02.343+00:00 Hi , @prathyusha chennareddy ,this is the recommended solution:
Error code: 4508
Message: Spark cluster not found.
Recommendation: Restart the debug session. -
prathyusha chennareddy 6 Reputation points
2023-01-06T13:33:42.07+00:00 Tried this but still facing the same issue
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 9%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKM%3C/text%3E%3C/svg%3E)
KranthiPakala-MSFT 46,737 Reputation points Microsoft Employee Moderator2023-01-11T00:00:42.523+00:00 Hello @prathyusha chennareddy ,
Thanks for the question and using MS Q&A platform.
As the initial error message says
User configuration issue, I suspect that the flow is not even entering the dataflow activity, it is failing at the very beginning of the dataflow activity execution. If that's the case, then it could be something related to a variable or a parameter value that you are trying to pass to your dataflow which is causing the issue.Could you please confirm if you are passing any values to your dataflow as parameters from your pipeline? If that is the case the I would recommend validating the parameter value w.r.t to the datatype of the dataflow parameter which might give you a hint what is causing the issue.
Do let us know how it goes.
-
KranthiPakala-MSFT 46,737 Reputation points Microsoft Employee Moderator
2023-01-12T22:31:46.2233333+00:00 Hello @prathyusha chennareddy ,
Following up to see if you have got a chance to see my previous response, if so, could you please validate your pipeline parameter value and the dataflow parameters data type and the value it is getting from pipeline parameter.
Do let us know how it goes.
Thank you
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(64, 12%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENM%3C/text%3E%3C/svg%3E)
Nirav Murarka 1 Reputation point2023-04-25T12:45:26.5533333+00:00 Hello @KranthiPakala-MSFT , We are facing similar issue while trying to run Data Flow in a For each loop. The activity runs fine for 2 item values in the loop, but fails in the 3rd run after running for 1 hour. In total the activity ran just over 4 hrs. We are using Azure IR here. Kindly help how to further debug the issue. Regards
Sign in to comment
Sign in to answer