Error while copying Files from Share Point Online to Azure Data Lake using Azure Data Factory?

Question

Hi,
I am trying to copy files from SPO to ADLS using ADF. I followed the instructions provided by MS : https://learn.microsoft.com/en-us/azure/data-factory/connector-sharepoint-online-list?tabs=data-factory#copy-file-from-sharepoint-online. The GetBearerToken activity runs perfectly and returns an access token but the Copy Activity fails with the following error:

ErrorCode=UserErrorInvalidCredentialToReadHttpFile,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The credential to read http file is invalid.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.Net.WebException,Message=The remote server returned an error: (401) Unauthorized.,Source=System,'

I have authorised the app for Full Control over SPO, enabled custom app authentication using SPO Management Shell and made sure that IRM is disabled.

I am not able to establish a connection to site/channelsite but the linked service connects to the communication site and downloads metadata.

Is there a way to resolve this issue?

-Thanks, Riyaz

Answer 1

Hello @Riyaz Budhwani ,
Thanks for the ask and using Microsoft Q&A platform .
As we understand the ask here is for a resolution to the above error . Please do let me know if that not accurate
To me it looks that the the application ID which you have created in Step#1 does not have the the required permission and so you are getting the error .
I request you to please revist the Step #2 of this document

Step #2
Grant SharePoint Online site permission to your registered application:
Note
This operation requires SharePoint Online site owner permission. You can find the owner by going to the site home page -> click the "X members" in the right corner -> check who has the "Owner" role.
Open SharePoint Online site link e.g. https://[your_site_url]/_layouts/15/appinv.aspx (replace the site URL).
Search the application ID you registered, fill the empty fields, and click "Create".
App Domain: localhost.com
Redirect URL: https://www.localhost.com
Permission Request XML:
XML
Copy
<AppPermissionRequests AllowAppOnlyPolicy="true">
<AppPermissionRequest Scope="http://sharepoint/content/sitecollection/web" Right="Read"/>
</AppPermissionRequests>
sharepoint grant permission
Click "Trust It" for this app.

Please do let me if you have any queries .
Thanks
Himanshu

-------------------------------------------------------------------------------------------------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

Answer 2

We face the same problem.
Any updates?

Answer 3

Hi @Riyaz Budhwani

I'm not sure if this topic is still relevant for you.

A few weeks ago I ran into the same problem and couldn't find an answer anywhere.

I was able to solve the problem by enabling Custom App Authentication. For SPO tenants that were created after 07.11.2018 the authentication via ACS app-only access token is disabled by default (see screenshot and link below).

security-apponly-azureacs

I decided to write a blog post with a step by step guide on how to move multiple files from an SPO folder to Blob Storage using Azure Data Factory, maybe you will find it usefull: copy-files-from-sharepoint-to-blob-storage-using-azure-data-factory