Copy files from Sharepoint to Azure Datalake using ADF

Anuja Ogale 1 Reputation point
2022-09-08T16:12:01.227+00:00
  1. I am trying to copy files from my sharepoint into the Azure Datalake and running into the following error in my copy activity.

"ErrorCode=HttpRequestFailedWithUnauthorizedError,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Http request failed with status code 401 Unauthorized, usually this is caused by invalid credentials, please check your activity settings."

  1. My Final goal is to copy all the files present in a particular folder and merge them in my sink. Is that possible with the HTTP source?

239100-screenshot-2022-09-08-120952.png

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,338 questions

2 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. Nandan Hegde 29,886 Reputation points MVP
    2022-09-08T17:55:31.883+00:00

    Can you please follow the steps documented here
    https://learn.microsoft.com/en-us/azure/data-factory/connector-sharepoint-online-list?tabs=data-factory#copy-file-from-sharepoint-online

    It contains all prerequisites for the access and creds

  2. Dimitri Bütikofer 1 Reputation point
    2022-10-27T09:16:33.487+00:00

    Hi @Anuja Ogale

    A few weeks ago I ran into the same problem following the documentation @Nandan Hegde recomended and couldn't find an answer anywhere.

    I was able to solve the problem by enabling Custom App Authentication. For SPO tenants that were created after 07.11.2018 the authentication via ACS app-only access token is disabled by default (see screenshot and link).

    254682-msft.png
    security-apponly-azureacs

    I decided to write a blog post with a step by step guide on how to move multiple files from an SPO folder to Blob Storage using Azure Data Factory, maybe you will find it usefull:
    copy-files-from-sharepoint-to-blob-storage-using-azure-data-factory

    0 comments