Azure synapse: Copy data activity completed successfully but no files were created in ADLS Gen2

Question

Azure synapse: Copy data activity completed successfully but no files were created in ADLS Gen2

Pravin007 1

Hi,

I have a pipeline that copies data from different data sources to ADLS. We run the pipeline (copy activity) and it executes successfully and in copy activity output its shows "file written 1" but when we check that file on ADLS, there is no file.

Do you know how to solve it?

Thanks in advance!

MartinJaffer-MSFT 26,236 Reputation points

2022-04-29T21:25:43.927+00:00

Hello @Pravin007 and welcome to Microsoft Q&A.

As I understand, the copy activity reports 1 written file, but you do not find the expected files?

There are a couple cases I can imagine something kinda similar could happen, but in all cases, something was written, even if not what or where expected.
This would be when parquet or partitioned files or subfolders are involved.

I recall one case where a file named _success was written, to indicate completion. This was a byproduct, not the main data. I suppose if there was no data, the success file would still be written.

Another possibility is it wrote to a subfolder, and the written file is in there. I think subfolder creation is one of the options when partitioning happens.

One more thing, some bits of ADLS can be hidden by certain viewers. Specifically the $logs container when logging is enabled on the storage account. It may be worth checking by another method.

Can you tell me how much data was written, not just the file count?
Pravin007 1 Reputation point

2022-05-02T12:15:28.13+00:00

Hi @MartinJaffer-MSFT ,

The copy data activity has written a 4.5Kb file.

In addition, if I rerun the same copy activity(copying the parquet files in the subfolders) at that time, the file has been created successfully in ADLS. It seems that this issue is intermittent.

Is there anything we can do about it?
MartinJaffer-MSFT 26,236 Reputation points

2022-05-03T19:48:12.073+00:00

There is most likely an edge case to explain the behavior, but if not, you can get a deeper investigation either by support ticket, or chat. If you go that route @Pravin007 , it is best to have the runIds of the pipeline runs in which the file is missing. Or their activityRunIds. Having these ID's let the support engineer find the appropriate logs quickly.

For self-investigation, I suggest looking at logs related to the storage account. If a file was indeed written, there should be a record of this.
Or perhaps do a search for any file with update timestamp near the completion of the Copy activity.

Data Factory overwrites files, unlike when the browser downloads and you get myDowload (2).txt . Could overwriting explain your cases?

Does your dataset have a parameterized name or filepath?
Pravin007 1 Reputation point

2022-05-04T13:47:49.357+00:00

Yes we have created the parametrized dataset. Please refer below screenshot.
Also I have checked the other folders also but not found the file.
MartinJaffer-MSFT 26,236 Reputation points

2022-05-10T20:30:53.567+00:00

Hmm doesn't look too complicated, and you don't seem to have the container parameterized... wait, there is a forward slash in container name. Maybe if that was moved to the next section?

I checked for parameterized container name because I was once able to put into the Copy activity logging a bad container name, and got no errors during run.

At this point I'm running out of ideas, @Pravin007 . Can I offer you a free 1-time support ticket so an engineer can look into the logs?

3 answers

Your answer

MartinJaffer-MSFT 26,236 Reputation points

2022-04-29T21:25:43.927+00:00

Hello @Pravin007 and welcome to Microsoft Q&A.

As I understand, the copy activity reports 1 written file, but you do not find the expected files?

There are a couple cases I can imagine something kinda similar could happen, but in all cases, something was written, even if not what or where expected.
This would be when parquet or partitioned files or subfolders are involved.

I recall one case where a file named _success was written, to indicate completion. This was a byproduct, not the main data. I suppose if there was no data, the success file would still be written.

Another possibility is it wrote to a subfolder, and the written file is in there. I think subfolder creation is one of the options when partitioning happens.

One more thing, some bits of ADLS can be hidden by certain viewers. Specifically the $logs container when logging is enabled on the storage account. It may be worth checking by another method.

Can you tell me how much data was written, not just the file count?
Pravin007 1 Reputation point

2022-05-02T12:15:28.13+00:00

Hi @MartinJaffer-MSFT ,

The copy data activity has written a 4.5Kb file.

In addition, if I rerun the same copy activity(copying the parquet files in the subfolders) at that time, the file has been created successfully in ADLS. It seems that this issue is intermittent.

Is there anything we can do about it?
MartinJaffer-MSFT 26,236 Reputation points

2022-05-03T19:48:12.073+00:00

There is most likely an edge case to explain the behavior, but if not, you can get a deeper investigation either by support ticket, or chat. If you go that route @Pravin007 , it is best to have the runIds of the pipeline runs in which the file is missing. Or their activityRunIds. Having these ID's let the support engineer find the appropriate logs quickly.

For self-investigation, I suggest looking at logs related to the storage account. If a file was indeed written, there should be a record of this.
Or perhaps do a search for any file with update timestamp near the completion of the Copy activity.

Data Factory overwrites files, unlike when the browser downloads and you get myDowload (2).txt . Could overwriting explain your cases?

Does your dataset have a parameterized name or filepath?
Pravin007 1 Reputation point

2022-05-04T13:47:49.357+00:00

Yes we have created the parametrized dataset. Please refer below screenshot.
Also I have checked the other folders also but not found the file.
MartinJaffer-MSFT 26,236 Reputation points

2022-05-10T20:30:53.567+00:00

Hmm doesn't look too complicated, and you don't seem to have the container parameterized... wait, there is a forward slash in container name. Maybe if that was moved to the next section?

I checked for parameterized container name because I was once able to put into the Copy activity logging a bad container name, and got no errors during run.

At this point I'm running out of ideas, @Pravin007 . Can I offer you a free 1-time support ticket so an engineer can look into the logs?

Answer 1

jhanns syr 6

edge or ASA, copy data makes it exasperating to copy one DLG2 path\dataFile to another DLG2 path\dataFile DLG2.
i have a fully functioning ASA Pipeline that should do this without issue. what i get is the source file rewritten in place, and the ASA copy data activity claiming success. but there is no success. there is no copy of the data file in the sink path as intended. source path, source file, sink path, sink file are all colocated on same ASA DLG2 data store. the only difference is source path and the sink path. they are distinctly different paths. this should not be. help from MS is requested.

jhanns syr 6 Reputation points

2023-04-10T19:50:54.9033333+00:00

not an answer; merely a closely related challenge.

Another possibility is it wrote to a subfolder, and the written file is in there. I think subfolder creation is one of the options when partitioning happens.

this is closely related as well; except the folder relationship isn't a 'subfolder' proper. see details below...

source <= '<root>\rootFolder\sourceDLG2path\sourceDLG2DataFile' sink <= '<root>\rootFolder\sinkDLG2path\sourceDLG2DataFile' // where sinkDLG2path <= concat('alternateFolder','',trim(string(sourceDLG2path)))

ASA copy file activity reports file copied successfully; metadata seems to confirm same; however, no sink copy of file is produced as described/intended. rather, the sourceDLG2DataFile timestamp reflects the actual source file was rewritten in place.

Answer 2

jhanns syr 6

believe it or not specifying one ASA 'dataset' for source and a second distinct ASA 'dataset' for sink got me past this. spooky

Answer 3

jhanns syr 6

Answer must be at least 10 characters. use separate ASA datasets for source, sink. that worked for me. spooky

Share via

Azure synapse: Copy data activity completed successfully but no files were created in ADLS Gen2

3 answers

not an answer; merely a closely related challenge.

Another possibility is it wrote to a subfolder, and the written file is in there. I think subfolder creation is one of the options when partitioning happens.

this is closely related as well; except the folder relationship isn't a 'subfolder' proper. see details below...

source <= '<root>\rootFolder\sourceDLG2path\sourceDLG2DataFile' sink <= '<root>\rootFolder\sinkDLG2path\sourceDLG2DataFile' // where sinkDLG2path <= concat('alternateFolder','',trim(string(sourceDLG2path)))

Your answer