I discovered that it was caused by a function used to create the path to write to. When I replaced it with a variable for the path, it worked as expected.
Error Code "6001"
I am running a Synapse Notebook through a synapse pipeline and am getting an error code 6001 "Creating new Synapse Notebook Statement Failed" and haven't been able to find anything explaining why. When running the code manually, it runs through to the final cell where it then is supposed to write to parquets in an azure data lake. Other notebooks are writing out fine, but this one doesn't seem to finish. It will write to parquets in a couple of minutes, and then just keeps spinning and never completes. Not sure if that is related to the issue received in when running in the pipeline.
The notebook is written in pyspark.