source files problem from internal server to Gen2 container

Question

source files problem from internal server to Gen2 container

Nick Wong 175

Hello,

I am trying to connect my interal files system to the gen2 container sink, following this instructions

https://arulmouzhi.wordpress.com/2021/04/12/bringing-folder-structure-via-azure-data-factory/

The first one in Allfiles set, I can load everything including folder and files.

User's image

When I follow the sourcebinary to be in the sink under the for loop copy file with file path @item().name, it seems to be no files or folders going into gen2 container at all, so I switch the sink from the file server instead of the URL shown as new sourcebinary but it shows

User's image

What wildcard should I use in order to avoid this problem from the attached

User's image

Many thanks

Nick

Anonymous

2024-05-31T15:33:12.16+00:00

@Nick Wong Thanks for using MS Q&A platform and posting your query.

To resolve this issue, you need to ensure that the source in your copy activity is pointing to a specific file or a set of files using wildcard characters, rather than a folder. Here’s what you can do:

Specify File Path: Instead of using @item().name which might be returning a folder path, ensure that the file path points directly to the files. For example, if you’re copying text files, your file path can be folderPath/*.txt.

Use Wildcards: If you want to copy all files within a folder, use the wildcard * to specify all files. For instance, folderPath/* would select all files within ‘folderPath’.

Check Dataset Configuration: In your dataset configuration for the source, make sure that the binary dataset is correctly configured to point to the files you intend to copy.

Pipeline Validation: Validate your pipeline again after making the changes to ensure that the error is resolved.

If you continue to face issues, please share more details about your pipeline configuration and the exact steps you’re following, so I can assist you further.

Remember, the binary copy feature is designed to copy file data as-is, without interpreting the file content. This means it expects a file or a list of files as input, not a directory.

Hope this helps. Do let us know if you any further queries.
Anonymous

2024-06-03T05:24:24.5466667+00:00

@Nick Wong We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Nick Wong 175 Reputation points

2024-06-03T11:28:37.16+00:00

@Anonymous Thank you for the answer, oh apparently, I tried different way and got the answer. When I dont fill anything inside the spaces in the source (file system) and sink (binary), it could transfer all the files over.
Anonymous

2024-06-03T11:57:08.8666667+00:00

@Nick Wong Glad to know your issue has been resolved. Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others "I'll repost your solution in case you'd like to accept the answer.

1 answer

Your answer

Anonymous

2024-05-31T15:33:12.16+00:00

@Nick Wong Thanks for using MS Q&A platform and posting your query.

To resolve this issue, you need to ensure that the source in your copy activity is pointing to a specific file or a set of files using wildcard characters, rather than a folder. Here’s what you can do:

Specify File Path: Instead of using @item().name which might be returning a folder path, ensure that the file path points directly to the files. For example, if you’re copying text files, your file path can be folderPath/*.txt.

Use Wildcards: If you want to copy all files within a folder, use the wildcard * to specify all files. For instance, folderPath/* would select all files within ‘folderPath’.

Check Dataset Configuration: In your dataset configuration for the source, make sure that the binary dataset is correctly configured to point to the files you intend to copy.

Pipeline Validation: Validate your pipeline again after making the changes to ensure that the error is resolved.

If you continue to face issues, please share more details about your pipeline configuration and the exact steps you’re following, so I can assist you further.

Remember, the binary copy feature is designed to copy file data as-is, without interpreting the file content. This means it expects a file or a list of files as input, not a directory.

Hope this helps. Do let us know if you any further queries.
Anonymous

2024-06-03T05:24:24.5466667+00:00

@Nick Wong We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Nick Wong 175 Reputation points

2024-06-03T11:28:37.16+00:00

@Anonymous Thank you for the answer, oh apparently, I tried different way and got the answer. When I dont fill anything inside the spaces in the source (file system) and sink (binary), it could transfer all the files over.
Anonymous

2024-06-03T11:57:08.8666667+00:00

@Nick Wong Glad to know your issue has been resolved. Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others "I'll repost your solution in case you'd like to accept the answer.

Answer 1

@Nick Wong - Welcome to Microsoft Q&A platform and thanks for posting your question.

I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others "I'll repost your solution in case you'd like to accept the answer.

Ask: I am trying to connect my interal files system to the gen2 container sink, following this instructions

https://arulmouzhi.wordpress.com/2021/04/12/bringing-folder-structure-via-azure-data-factory/

The first one in Allfiles set, I can load everything including folder and files.

User's image

When I follow the sourcebinary to be in the sink under the for loop copy file with file path @item().name, it seems to be no files or folders going into gen2 container at all, so I switch the sink from the file server instead of the URL shown as new sourcebinary but it shows

User's image

What wildcard should I use in order to avoid this problem from the attached

User's image

Solution: When I dont fill anything inside the spaces in the source (file system) and sink (binary), it could transfer all the files over.

If I missed anything please let me know and I'd be happy to add it to my answer, or feel free to comment below with any additional information.

If you have any other questions, please let me know. Thank you again for your time and patience throughout this issue.

Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.

Anonymous

2024-06-04T10:27:04.45+00:00

@Nick Wong Following up to see if the above answer was helpful. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Share via

source files problem from internal server to Gen2 container

1 answer

Your answer