Ask for using Azure Data Factory to transfer files from SharePoint to Azure Blob Storage

Question

Hi, I would like to ask about Azure Data Factory’s capabilities for migrating large files from SharePoint to Azure Blob Storage. Does ADF support preserving the original file names, folder hierarchy, file owners, basic metadata, and Entra ID access permissions during the migration?”

Answer 1

Hi @Bui Quoc Bao

Hey Bui Quoc Bao! It sounds like you're looking to use Azure Data Factory (ADF) for transferring large files from SharePoint to Azure Blob Storage while maintaining certain attributes like file names, folder hierarchy, owners, and permissions.

Here's what you should know:

File Copying: Azure Data Factory supports copying files from SharePoint to Azure Blob Storage using pipelines. You typically use the Copy Activity and may need to configure the source as SharePoint and the sink as Azure Blob Storage.

File Names and Folder Hierarchy: When transferring files, ADF should preserve the original file names. However, ensuring that folder hierarchy is maintained might require additional configurations, such as using folder paths in your dataset settings.

File Owners and Metadata: As for metadata like file owners, that's not natively transferred when using ADF. You may need to implement a separate workflow to store this information if it’s critical for your use case.

Access Permissions: Copying files won't automatically preserve access permissions (like Entra ID permissions). You might need to explicitly set access rights on the destination storage after the transfer.

For your specific needs, here are a few steps you can follow:

Set Up Linked Services: Create linked services (connections) for both your SharePoint and Azure Blob Storage.

Use Copy Activity: Set up the Copy Activity in your ADF pipeline to fetch files from SharePoint and transfer them to Blob Storage. Make sure to set the necessary parameters in your dataset to reflect the original file structure.

Metadata Management: Consider writing custom code or using logic apps to handle metadata and permissions post-transfer, depending on your requirements.

If you face any issues, or if you need to confirm the capabilities regarding metadata and permissions, let’s gather some more details to assist you better:

Follow-Up Questions:

What specific file types are you transferring from SharePoint? Are there any limitations on file sizes that you need to manage? Can you clarify what specific metadata and permissions you require to maintain during the transfer? Have you already set up the necessary permissions in SharePoint for the ADF service principal to perform the file operations? Are there any compliance or security features you need to consider for the data you're transferring? Feel free to reach out if you have any more questions or need further assistance!

References:

Copy data from SharePoint Online List Azure Data Factory Documentation

• SharePoint Online Permissions Guide
• Move Files with Azure Data Factory

Answer 2

Azure Data Factory (ADF) can be used to transfer files from SharePoint to Azure Blob Storage, but the specific capabilities regarding preserving original file names, folder hierarchy, file owners, basic metadata, and Entra ID access permissions during the migration are not detailed in the provided context.

However, ADF does support preserving the file hierarchy in the target folder through its copyBehavior property, which can be set to PreserveHierarchy. This means that the relative path of the source file to the source folder will be identical to the relative path of the target file to the target folder.

As for file owners, basic metadata, and Entra ID access permissions, the context does not provide specific information on whether these attributes are preserved during the migration process. You may need to consult additional Azure documentation or resources for detailed capabilities regarding metadata and permissions preservation during file transfers.