Error when running pyspark on synapse notebook - Type with name 'SQLAnalyticsConnectorDataSourcec1df77c5c38e42aea523b511df53d648' already exists.

DivyaK-4075 11 Reputation points
2022-10-28T10:45:33.87+00:00

Hi,

I am getting this error when trying to write the dataframe to storage. The job was running fine all these day's, but all of sudden its failing with below error, please can you let me know what could be the issue:

---------------------------------------------------------------------------  
Py4JJavaError                             Traceback (most recent call last)  
<ipython-input-19-1b559d1c> in <module>  
      1 # Write spark dataframe as a csv file  
      2 csv_path = adls_path + clientCode + '/processed/'+SqlPoolName + '/'+TableName+'_nlp_classlabel.csv'  
----> 3 unioned_df.repartition(1).write.csv(csv_path, mode = 'overwrite', header = 'true')  
      4   
      5 # Write duplicate ticket descriptions as a csv file  
  
/opt/spark/python/lib/pyspark.zip/pyspark/sql/readwriter.py in csv(self, path, mode, compression, sep, quote, escape, header, nullValue, escapeQuotes, quoteAll, dateFormat, timestampFormat, ignoreLeadingWhiteSpace, ignoreTrailingWhiteSpace, charToEscapeQuoteEscaping, encoding, emptyValue, lineSep)  
   1370                        charToEscapeQuoteEscaping=charToEscapeQuoteEscaping,  
   1371                        encoding=encoding, emptyValue=emptyValue, lineSep=lineSep)  
-> 1372         self._jwrite.csv(path)  
   1373   
   1374     def orc(self, path, mode=None, partitionBy=None, compression=None):  
  
~/cluster-env/clonedenv/lib/python3.8/site-packages/py4j/java_gateway.py in __call__(self, *args)  
   1302   
   1303         answer = self.gateway_client.send_command(command)  
-> 1304         return_value = get_return_value(  
   1305             answer, self.gateway_client, self.target_id, self.name)  
   1306   
  
/opt/spark/python/lib/pyspark.zip/pyspark/sql/utils.py in deco(*a, **kw)  
    109     def deco(*a, **kw):  
    110         try:  
--> 111             return f(*a, **kw)  
    112         except py4j.protocol.Py4JJavaError as e:  
    113             converted = convert_exception(e.java_exception)  
  
~/cluster-env/clonedenv/lib/python3.8/site-packages/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)  
    324             value = OUTPUT_CONVERTER[type](answer[2:], gateway_client)  
    325             if answer[1] == REFERENCE_TYPE:  
--> 326                 raise Py4JJavaError(  
    327                     "An error occurred while calling {0}{1}{2}.\n".  
    328                     format(target_id, ".", name), value)  
  
Py4JJavaError: An error occurred while calling o1396.csv.  
: java.sql.SQLException: com.microsoft.sqlserver.jdbc.SQLServerException: Type with name 'SQLAnalyticsConnectorDataSourcec1df77c5c38e42aea523b511df53d648' already exists.  

Thanks,
Divya

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,565 questions
{count} votes