Synapse notebook not working from pipeline

Blair 31 Reputation points

I have a Notebook in Azure Synapse written in spark (Scala).
It uses the synapsesql function to read data from a SQL pool and to push data into a SQL pool.

When I run the notebook everything works perfectly and the table in the SQL pool is created and populated. However when I call the notebook from a Synapse pipeline it succeeds but doesn't do anything

Any ideas?

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
3,791 questions
{count} votes

1 answer

Sort by: Most helpful
  1. AnnuKumari-MSFT 27,926 Reputation points Microsoft Employee

    Hi @Blair ,

    Thankyou for using Microsoft Q&A platform and posting your query.

    Looking at your query, it seems when you execute the notebook directly , you are able to see the output . However, when you call the same notebook via Synapse pipelines , pipeline gets succeeded but output doesn't show up .

    It is an expected behavior. To view the output after pipeline execution ,

    • Pick up the Apache Spark application name from the output of pipeline
    • Navigate to Apache Spark Application under Monitor tab and search for the same application name .
    • These 4 tabs would be available there: Diagnostics,Logs,Input data,Output data
    • Go to Logs ad check 'stdout' for getting the required output.

    Attaching video for your reference:

    For more details, please refer to this video:

    If this answers your query, please do let us know.