ADF Pipeline with ADB notebook hasn't written files to ADLS

Ezeabasili, Remmy 1 Reputation point
2021-12-03T10:34:58.293+00:00

HI,

I have an ADF piepline which contains an Azure Databricks notebook. The notebook contains multiple commands which write a spark dataframe to a single file in ADLS.
When triggering the pipeline the pipeline was able to run most of the commands but has erred on the penultimate command. The piepline failed with error ref 3204.
As such, I thought that the majority of files would have been written to ADLS, but when I checked, they were not there. Could someone explain why this is?

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,270 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,843 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,059 questions
{count} votes

1 answer

Sort by: Most helpful
  1. HimanshuSinha-msft 19,281 Reputation points Microsoft Employee
    2021-12-03T22:28:15.503+00:00

    Hello @Ezeabasili, Remmy ,
    Thanks for the ask and using Microsoft Q&A platform .
    When you run the notebook does it also fail ? When we run a notebook from ADF , ADF does only work as trigger and initiate the notebook . I think
    Error code 3204 , stands for "Message: Job execution failed." and so the cluster logs should have more helpful error . What is the code doing the "penultimate command" , can you please share that .
    Moreover if it is an intermittent issue and is not happening always , may be you can set the Retry and retry interval .

    154790-image.png

    Please do let me know how it goes .
    Thanks
    Himanshu

    -------------------------------------------------------------------------------------------------------------------------

    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
    • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators
    0 comments No comments