How do I copy data from a public sharepoint URL to Microsoft Blob Storage?

Question

Hi everyone!

I am new to using Azure services. My co-author at Stanford has uploaded several files (in csv format) to a public link. This link starts as: office365stanford-my.sharepoint.com/personal

It has 40 files, ranging from a few 100 MB to 100 GB. I don't want to have to download the files to my local machine to transfer it to Blob. How can I do this directly?

I tried to set up Azure Data Factory but the pipleline is failing with the following error:

ErrorCode=HttpRequestFailedWithClientError,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Http request failed with client error, status code 403 Forbidden, please check your activity settings. If you configured a baseUrl that includes path, please make sure it ends with '/'.

Request URL: https://office365stanford-my.sharepoint.com/personal/gxzheng_stanford_edu/_layouts/15/onedrive.aspx?id=%2Fpersonal%2Fgxzheng_stanford_edu%2FDocuments%2FCNpatents%2Ffull_text_en&ga=1/.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.Net.WebException,Message=The remote server returned an error: (403) Forbidden.,Source=System,'

Note, I have already set up Microsoft Blob and I'm able to use AzCopy to transfer my local files into it.

Thanks for all your help!!

Answer 1

You can use Copy Blob (programmatically) - more at https://stackoverflow.com/questions/45003502/save-an-image-from-url-to-azure-storage-directly?noredirect=1&lq=1 or use tools such as this one https://docs.fineuploader.com/branch/master/features/azure.html

---

If the above response helps answer your question, remember to "Accept Answer" so that others in the community facing similar issues can easily find the solution. Your contribution is highly appreciated.

hth

Marcin

Answer 2

Here are some alternative methods to transfer your files directly:

• Azure Logic Apps: You can create an Azure Logic App that automates the file transfer from SharePoint to Azure Blob Storage.
• Power Automate: Microsoft Power Automate can also be used to copy files from SharePoint to Azure Blob Storage, though it may have limitations on the file size.
• Server-to-Server Transfer: Utilize AzCopy’s server-to-server transfer capabilities to move files directly between storage servers.

Additional Reading: https://learn.microsoft.com/en-us/answers/questions/1513111/how-to-copy-folders-and-files-from-a-sharepoint-si

Answer 3

Hello @Namrata Narain

It seems like you are trying to transfer files from a public link to Azure Blob Storage without downloading them to your local machine. One way to do this is by using the Azure Data Factory. However, since you are facing an error with the pipeline, it might be helpful to check the activity settings and ensure that the baseUrl ends with '/'.

Another way to transfer files directly from a public link to Azure Blob Storage is by using the Azure Storage Explorer. You can download and install the Azure Storage Explorer on your local machine and then use it to transfer files from the public link to Azure Blob Storage.

Here are the steps to do so:

1. Open the Azure Storage Explorer and connect to your Azure Blob Storage account.
2. Right-click on the container where you want to transfer the files and select "Upload".
3. In the "Upload Blob" dialog box, select "From URL" and enter the URL of the file you want to transfer.
4. Enter the name of the blob you want to create in the container and click "Upload". You can repeat these steps for each file you want to transfer. This method allows you to transfer files directly from the public link to Azure Blob Storage without downloading them to your local machine. I hope this helps

---

I hope that this response has addressed your query and helped you overcome your challenges. If so, please mark this response as Answered. This will not only acknowledge our efforts, but also assist other community members who may be looking for similar solutions.