Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,287 questions
ADF: Job failed due to reason: at Sink 'finalSink': Error while reading file abfss:REDACTED_LOCAL_PART
Brand Johannes
1
Reputation point
Hello,
I got an error every time, my data flow in Azure Data Factory is triggered writing:
Job failed due to reason: at Sink 'finalSink': Error while reading file abfss:REDACTED_LOCAL_PART@xyz .dfs.core.windows.net/CustomerInsights_xyzzyx/System_Profile/Customer/2022/08/24/1243/Customer_5.csv.
The data flow runs without any issues in the debug mode and also the file raising the error differs from time to time.
How can I fix this problem with this little information?
Best,
Johannes
{count} votes
-
ShaktiSingh-MSFT 5,761 Reputation points Microsoft Employee2022-08-29T15:34:43.697+00:00 Hi @Brand Johannes ,
Welcome to Microsoft Q&A platform and thanks for using Azure services.
As I understand your question, you are using Azure Data Factory where Source is a csv file, where Pipeline succeeds in debug mode but during Triggered run it throws
redacted_local_parterror.
Please correct me if I understood it differently.For this error, I would suggest you to try
Cached Sinkonce and let us know the result.
Detailed link for implementation of cache at sink are here:Please let us know if you face any issue.
Awaiting your response. Thank you!
Hey,
you are mostly correct. I read multiple Inline datasets and want to merge them into one common record. Unfortunately the dataflow always fails with the REDACT_LOCAL_PART error.
This is what I get, when I set the only sink to cached. Still the same error, but now with a different file.
Anything that could make the flow proceed?
Best,
Johannes
Hi @Brand Johannes ,
Sorry to hear that you are facing this error.
In order to troubleshoot this issue in a better way, could you please share below details with us:
Thanks for your patience, looking forward for you reply.
Hi @Brand Johannes ,
Just checking if you could you please share below details with us:
Pipeline Screenshot
Activities Used
Dataset settings
Source and Sink
Dummy data
Any other relevant details pertaining to this issue.
Thanks for your patience, looking forward for you reply.
Sign in to comment