ADF Pipeline with ADB notebook hasn't written files to ADLS

Question

HI,

I have an ADF piepline which contains an Azure Databricks notebook. The notebook contains multiple commands which write a spark dataframe to a single file in ADLS.
When triggering the pipeline the pipeline was able to run most of the commands but has erred on the penultimate command. The piepline failed with error ref 3204.
As such, I thought that the majority of files would have been written to ADLS, but when I checked, they were not there. Could someone explain why this is?

Answer 1

Hello @Ezeabasili, Remmy ,
Thanks for the ask and using Microsoft Q&A platform .
When you run the notebook does it also fail ? When we run a notebook from ADF , ADF does only work as trigger and initiate the notebook . I think
Error code 3204 , stands for "Message: Job execution failed." and so the cluster logs should have more helpful error . What is the code doing the "penultimate command" , can you please share that .
Moreover if it is an intermittent issue and is not happening always , may be you can set the Retry and retry interval .

Please do let me know how it goes .
Thanks
Himanshu

-------------------------------------------------------------------------------------------------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators