Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,311 questions
ADF: Job failed due to reason: at Sink 'finalSink': Error while reading file abfss:REDACTED_LOCAL_PART
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(150.4, 79%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBJ%3C/text%3E%3C/svg%3E)
Brand Johannes
1
Reputation point
Hello,
I got an error every time, my data flow in Azure Data Factory is triggered writing:
Job failed due to reason: at Sink 'finalSink': Error while reading file abfss:REDACTED_LOCAL_PART@xyz .dfs.core.windows.net/CustomerInsights_xyzzyx/System_Profile/Customer/2022/08/24/1243/Customer_5.csv.
The data flow runs without any issues in the debug mode and also the file raising the error differs from time to time.
How can I fix this problem with this little information?
Best,
Johannes
{count} votes
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(220.8, 4%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESM%3C/text%3E%3C/svg%3E)
ShaktiSingh-MSFT 16,171 Reputation points2022-08-29T15:34:43.697+00:00 Hi @Brand Johannes ,
Welcome to Microsoft Q&A platform and thanks for using Azure services.
As I understand your question, you are using Azure Data Factory where Source is a csv file, where Pipeline succeeds in debug mode but during Triggered run it throws
redacted_local_parterror.
Please correct me if I understood it differently.For this error, I would suggest you to try
Cached Sinkonce and let us know the result.
Detailed link for implementation of cache at sink are here:Please let us know if you face any issue.
Awaiting your response. Thank you! -
Brand Johannes 1 Reputation point
2022-08-31T13:57:49.073+00:00 Hey,
you are mostly correct. I read multiple Inline datasets and want to merge them into one common record. Unfortunately the dataflow always fails with the REDACT_LOCAL_PART error.
This is what I get, when I set the only sink to cached. Still the same error, but now with a different file.
Job failed due to reason: at Sink 'finalSink': Error while reading file abfss:REDACTED_LOCAL_PART@xyz .dfs.core.windows.net/CustomerInsights_xyzzyx/System_Profile/Customer/2022/08/24/1243/Customer_2.csv.
Anything that could make the flow proceed?
Best,
Johannes -
ShaktiSingh-MSFT 16,171 Reputation points
2022-09-01T17:02:01.087+00:00 Hi @Brand Johannes ,
Sorry to hear that you are facing this error.
In order to troubleshoot this issue in a better way, could you please share below details with us:
- Pipeline Screenshot
- Activities Used
- Dataset settings
- Source and Sink
- Dummy data
- Any other relevant details pertaining to this issue.
Thanks for your patience, looking forward for you reply.
-
ShaktiSingh-MSFT 16,171 Reputation points
2022-09-05T09:07:58.817+00:00 Hi @Brand Johannes ,
Just checking if you could you please share below details with us:
Pipeline Screenshot
Activities Used
Dataset settings
Source and Sink
Dummy data
Any other relevant details pertaining to this issue.
Thanks for your patience, looking forward for you reply.
Sign in to comment
Sign in to answer