Error when running pyspark on synapse notebook - Type with name 'SQLAnalyticsConnectorDataSourcec1df77c5c38e42aea523b511df53d648' already exists.

Question

Hi,

I am getting this error when trying to write the dataframe to storage. The job was running fine all these day's, but all of sudden its failing with below error, please can you let me know what could be the issue:

---------------------------------------------------------------------------  
Py4JJavaError                             Traceback (most recent call last)  
<ipython-input-19-1b559d1c> in <module>  
      1 # Write spark dataframe as a csv file  
      2 csv_path = adls_path + clientCode + '/processed/'+SqlPoolName + '/'+TableName+'_nlp_classlabel.csv'  
----> 3 unioned_df.repartition(1).write.csv(csv_path, mode = 'overwrite', header = 'true')  
      4   
      5 # Write duplicate ticket descriptions as a csv file  
  
/opt/spark/python/lib/pyspark.zip/pyspark/sql/readwriter.py in csv(self, path, mode, compression, sep, quote, escape, header, nullValue, escapeQuotes, quoteAll, dateFormat, timestampFormat, ignoreLeadingWhiteSpace, ignoreTrailingWhiteSpace, charToEscapeQuoteEscaping, encoding, emptyValue, lineSep)  
   1370                        charToEscapeQuoteEscaping=charToEscapeQuoteEscaping,  
   1371                        encoding=encoding, emptyValue=emptyValue, lineSep=lineSep)  
-> 1372         self._jwrite.csv(path)  
   1373   
   1374     def orc(self, path, mode=None, partitionBy=None, compression=None):  
  
~/cluster-env/clonedenv/lib/python3.8/site-packages/py4j/java_gateway.py in __call__(self, *args)  
   1302   
   1303         answer = self.gateway_client.send_command(command)  
-> 1304         return_value = get_return_value(  
   1305             answer, self.gateway_client, self.target_id, self.name)  
   1306   
  
/opt/spark/python/lib/pyspark.zip/pyspark/sql/utils.py in deco(*a, **kw)  
    109     def deco(*a, **kw):  
    110         try:  
--> 111             return f(*a, **kw)  
    112         except py4j.protocol.Py4JJavaError as e:  
    113             converted = convert_exception(e.java_exception)  
  
~/cluster-env/clonedenv/lib/python3.8/site-packages/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)  
    324             value = OUTPUT_CONVERTER[type](answer[2:], gateway_client)  
    325             if answer[1] == REFERENCE_TYPE:  
--> 326                 raise Py4JJavaError(  
    327                     "An error occurred while calling {0}{1}{2}.\n".  
    328                     format(target_id, ".", name), value)  
  
Py4JJavaError: An error occurred while calling o1396.csv.  
: java.sql.SQLException: com.microsoft.sqlserver.jdbc.SQLServerException: Type with name 'SQLAnalyticsConnectorDataSourcec1df77c5c38e42aea523b511df53d648' already exists.  

Thanks,
Divya