Not able to run spark pool notebook from synapse pipeline

Question

Hi,

I am having a spark notebook in azure synapse .

about pipeline : I am reading data from dedicated synapse pool (It could be either in same workspace or in some other workspace ) and write it to a destination

notebook works fine standalone but when we try to run it through pipeline it fails giving below error :

"errorCode": "6002",
"message": "com.microsoft.spark.sqlanalytics.exception.SQLAnalyticsConnectorException: jdbc:<server>.sql.azuresynapse.net:1433;database=dataeng;encrypt=true;trustServerCertificate=false;loginTimeout=300;\nLogin failed for user '<token-identified principal>'. ClientConnectionId:<Id>\n\n at com.microsoft.spark.sqlanalytics.utils.Utils$.createConnection(Utils.scala:344)\n\n at com.microsoft.spark.sqlanalytics.read.SQLAnalyticsReader.com$microsoft$spark$sqlanalytics$read$SQLAnalyticsReader$$connection$lzycompute(SQLAnalyticsReader.scala:39)\n\n at com.microsoft.spark.sqlanalytics.read.SQLAnalyticsReader.com$microsoft$spark$sqlanalytics$read$SQLAnalyticsReader$$connection(SQLAnalyticsReader.scala:39)\n\n at com.microsoft.spark.sqlanalytics.read.SQLAnalyticsReader.readSchema(SQLAnalyticsReader.scala:82)\n\n at org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation$.create(DataSourceV2Relation.scala:175)\n\n at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:224)\n\n at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:187)\n\n at org.apache.spark.sql.SqlAnalyticsConnector$SQLAnalyticsFormatReader.sqlanalytics(SqlAnalyticsConnector.scala:45)\n\n at org.apache.spark.sql.SqlAnalyticsConnector$SQLAnalyticsFormatReader.synapsesql(SqlAnalyticsConnector.scala:28)\n\n ... 56 elided\n",
"failureType": "UserError",
"target": "Notebook1",
"details": []

==================

Answer 1

Hello @Thakur, Abhishek

Please refer to a similar question here

Let me know of any queries