Copying files from sharepoint to blob

Question

Hello @Harishga I have three CSV files in the SharePoint folder (Shared Documents/Test/Invoices) that I want to copy to a blob storage, orders/invoices/PreProcessedinvoices

In the web activity for "Get SPO Folder Metadata," I have the URL below, which I am just getting the metadata for the files in the Invoices folder.

https://dreamfieldsinc.sharepoint.com/sites/InformationSystems/_api/web/GetFolderByServerRelativeUrl('/sites/InformationSystems/Shared Documents/Test/Invoices')/Files
In the source I have used a Binary dataset,
base URL - https://dreamfieldsinc.sharepoint.com
relative URL- /sites/InformationSystems/_api/web/GetFolderByServerRelativeUrl('/sites/InformationSystems/Shared Documents/Test/Invoices')/Files

In the sink dataset, I am also using a binary dataset and I have pointed that to the blob storage
But the issue is when I run the pipeline, I see the sites folder instead of the three files being copied, is there a solution around this to copy the actual files in the folder instead of the entire sites folder? Is there something I am missing here? I specified the Files at the end of the relative URL but that does not seem to work. Please help @Harishga

Answer 1

Hi @Kayode Ogidan
Welcome to Microsoft Q&A platform and thanks for posting your question here.

To copy individual CSV files from a SharePoint folder to Azure Blob Storage, you need to make sure that your data pipeline is set up correctly. Here are some steps you can take to troubleshoot and resolve the issue:

• Check that the 'Get SPO Folder Metadata' web activity is returning a list of file metadata, including the relative URLs for each file in the SharePoint folder.
• Make sure that the 'Foreach' loop is set up to iterate over the array of file metadata and that it references the output of the 'Get SPO Folder Metadata' activity correctly.
• Use dynamic content in the copy activity to specify the path for each file within the 'Foreach' loop.
• Confirm that the sink dataset pointing to the blob storage is configured to receive individual files and that the path is dynamic based on the file being processed in the current iteration of the loop.
• Review the pipeline run logs to identify any errors or misconfigurations.
• As a troubleshooting step, try configuring the pipeline to copy just one file from the SharePoint folder to the blob storage. This can help identify the issue and ensure that the basic copy operation works as expected.

I hope this information helps you. Let me know if you have any further questions or concerns.