ADF copy-activity from Microsoft 365 to storage account fails

Yonatan Shlain 0 Reputation points
2024-06-09T22:26:35.5233333+00:00

Hello,

I’ve a continuous problem with my ADF pipeline - I’m trying to run a "copy-data" pipeline and encounter access and permission errors with my specific resources.

  • My source is a Microsoft 365 Table connector (I’m retrieving some columns from my organization mails)
  • My sink is a storage account.

In my SA access control properties, I’ve given my app appropriate IAM role with the all the necessary permissions so it can access the storage account and write the data successfully.

Now since I don’t want my storage to be public, I chose the public network access to be "Enabled from selected virtual networks and IP addresses".

First try:

Since ADF is a resource instance in my Azure subscription, I’ve specified my instance to have access to my storage account based on its system-assigned managed identity (Microsoft.DataFactory/factories) and configured all propely (see https://roshan-vin4u.medium.com/authenticate-azure-data-factory-with-azure-data-lake-gen-2-using-managed-identities-3663f1449440).

But when I ran the pipeline it failed, claiming I can’t use system-assigned managed identity with Microsoft 365 connector.

Second try:

I tried to access my storage account with private link using ADF PE, so I've created private endpoint and configured it all properly (see https://learn.microsoft.com/en-us/answers/questions/635312/connect-data-factory-to-azure-storage-wiht-private).

First I configured the service endpoint to be the storage's dfs url and run the pipeline, got the error:

"ErrorCode=UserErrorOffice365DataLoaderError,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Office365 data loading failed to execute. office365LoadErrorType: PermanentError ...Failure happened on 'Sink' side. ErrorCode=AdlsGen2ForbiddenError"

Then I configured it to be the storage's blob url and got the next error "the remote server return an error: (403) .... Unable to create Azure blob container"

Conclusions

I've also tested it and got same results when disabling public network access.

My conclusion is that the runtime did access the storage account via the private endpoint but failed (either due to some misconfiguration or some functionality problem).

The weirdest issue is when I tried using a simple ADF copy-data pipeline from one storage account to another using PE for my linked-service and integration runtime, it did ran successfully.

What can be the problem? How do I solve this issue?

Thank you!

Microsoft 365
Microsoft 365
Formerly Office 365, is a line of subscription services offered by Microsoft which adds to and includes the Microsoft Office product line.
4,064 questions
Azure Storage Accounts
Azure Storage Accounts
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
2,811 questions
Azure Private Link
Azure Private Link
An Azure service that provides private connectivity from a virtual network to Azure platform as a service, customer-owned, or Microsoft partner services.
479 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,862 questions
{count} votes