Call notebook from Spark job

Question

Call notebook from Spark job

Ryan Abbey 1,186

Something I didn't think would be particularly complex but no details of how to do such in the MS help files apache-spark-job-definitions

I've created a Spark notebook that I want to call from a Spark job definition, how to do such? Or am I overthinking and should follow exactly how it's done for DataBricks?

1 answer

Your answer

Answer 1

HimanshuSinha-msft 19,486 Microsoft Employee Moderator

Hello @Ryan Abbey ,
Thanks for the ask and using the Microsoft Q&A platform .
You can do it from Synapse Studio . the below screen shot should help .

Please do let me know how it goes .
Thanks
Himanshu
Please do consider clicking on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members

Ryan Abbey 1,186 Reputation points

2021-08-10T03:36:33.92+00:00

Yes, it's known that I can run a notebook from a pipeline but I want to be able to run a notebook from a Spark job
HimanshuSinha-msft 19,486 Reputation points Microsoft Employee Moderator

2021-08-10T03:52:52.383+00:00

Thanks @Ryan Abbey for clarifying and I don't think at this you can run a notebook from the spark job . But I am just curiuos as to what is the miss if I trigger a notebook from a pipeline then if I do the same from Spark job . I am sure I am missing out something here .

Yes in Adb we can just browse to the notebook while creating the spark job ,
Thanks
Himanshu
Ryan Abbey 1,186 Reputation points

2021-08-10T20:21:17.417+00:00

I'd say what's the point of a spark job if it can't kick off a spark notebook... having to create functionality outside of Synapse, upload it and repeat until we get settings right is pretty ludicrous... all the more so because we can't seem to pass parameters from Logic Apps to it from a REST API call...

Share via

Call notebook from Spark job

1 answer

Your answer