Tutorial For Migrating Content From Azure Blob Storage Container To SharePoint Embedded Container

Purpose

This tutorial will guide you through migrating content from Azure Blob Storage (ABS) to SharePoint Embedded (SPE) using C#. This is useful for customers who have 500 docs in the blob storage container.

Prerequisites

A Microsoft Entra ID application registration. See register an application.
Your Microsoft Entra ID tenant has a Microsoft 365 subscription.
A Microsoft Entra ID tenant. If you don't have a tenant, create a free Azure account to get a free subscription.
An account with at least the Global Administrator or SharePoint Embedded Administrator role.
.NET Core SDK version 8.0.303
Dotnet environment to run the sample app
- It can be run on Windows, Linux and macOS
SharePoint Embedded container
- For more information on how to set up a SPE container
Azure Blob Storage container
- For more information on how to set up an ABS container

Authentication

Azure Blob Storage

Credentials - Container-level Shared Access Signature (SAS) URL.
Permission - Read and List

SharePoint Embedded

An Azure account
A SharePoint Tenant where you'll create your containers and its Tenant ID
An onboarded application ID (sometimes called client ID) and its corresponding ContainerTypeId
Create a new App Registration in Microsoft Entra ID portal.
In the App Registration, add a new Mobile & Console application platform in Microsoft Entra ID App Registration Authenticate portal
A ContainerType
A Container
Having the application registered in the consuming tenant (even if the owner of the application is the same as the consuming)
Having the containerType registered in the consuming tenant (even if the owner of the CT is the same as the consuming)
Consuming tenant user name and password credentials - will be required to authenticate the Microsoft Graph client
Permission - "User.Read", "FileStorageContainer.Selected"

Migrating Data from Azure Blob Storage container to SharePoint Embedded container

Description

This section provides code snippets on how to accomplish the migration. All the validation has been removed for readability.

Connecting to Azure Blob Storage Container

_containerClient = new BlobContainerClient(new Uri(_containerLevelSASUrl));

Connecting to SharePoint Embedded

string[] _scopes = { "User.Read", "FileStorageContainer.Selected" };
InteractiveBrowserCredentialOptions interactiveBrowserCredentialOptions = new InteractiveBrowserCredentialOptions()
  {
    ClientId = clientId,
    RedirectUri = new Uri("http://localhost"),
  };
InteractiveBrowserCredential interactiveBrowserCredential = new InteractiveBrowserCredential(interactiveBrowserCredentialOptions);

_graphClient = new GraphServiceClient(interactiveBrowserCredential, scopes, null);

// Will open up a browser to provide your consuming tenant admin credentials
var user = await _graphClient.Me.GetAsync();

Getting the blob list

var blobs = new List<string>();
await foreach (var blobItem in _containerClient.GetBlobsAsync())
{
  blobs.Add(blobItem.Name);
}
return blobs;

Thread pooling

private CountdownEvent _countdown;

// This is how the thread pool knows how many files are being migrated
_countdown = new CountdownEvent(blobs.Count);

FileStructure

public class FileStructure
{
  public string blobName { get; set; }
  public string parentFolderId { get; set; }
}

Traverse blob list

// It creates a new folder in the destination. The name of the folder is the blob's container name.
// root means it is the root of the document library.
// If you want to copy it to another drive item, you can put the drive item ID here.
containerFolder = await _graphClient.CreateFolder(_containerName, "root");

// Traverse the blob list
foreach (var blobName in fileList)
{
  FileStructure fs = new FileStructure() { blobName = blobName };

  // This function parses the flat file into the folder hierarchy and creates the folder structure in the destination. It will retrieve the parentFolderId that the file should be copied to.
  // If you are going to copy it to root you can comment this line out. The parentFolderId will be containerFolder.Id
  fs.parentFolderId = TraverseBlobName(fs, containerFolder.Id)

  // This is where the thread pool happens.
  // It takes in a callback function and an Object parameter.
  ThreadPool.QueueUserWorkItem(MigrateFile, fs);
}

// Call so the program doesn't end, it waits for all the files to be processed
_countdown.Wait();

Traverse blob name

// Parse for folder path not including the file name and put it in an array
var pathSegments = filePath.Split(new char[] { '/' }, StringSplitOptions.RemoveEmptyEntries);
string[] directoriesParts = pathSegments.Take(pathSegments.Length - 1).ToArray();

// Traverse the folder listing and create 1 folder at a time
string relativePath = _containerName;
string newFolderId = parentFolderId;
foreach (string folderName in directoriesParts)
{
  string newPath = relativePath + _separator + folderName;
  ...

  DriveItem subFolder = await _graphClient.CheckIfItemExists(folderName, newFolderId);
  if (subFolder == null)
  {
    subFolder = await _graphClient.CreateFolder(folderName, newFolderId);
    ...
  }
  newFolderId = subFolder.Id;

  relativePath = newPath;
}

return newFolderId;

Check if the item exists

var item = await _graphClient.Drives[_containerId].Items[parentFolderId].ItemWithPath(itemPath).GetAsync();

Create folder

var folder = new DriveItem
{
  Name = folderName,
  Folder = new Folder(),
  AdditionalData = new Dictionary<string, object>()
  {
    { "@microsoft.graph.conflictBehavior", "fail" }
  }
};
var createdFolder = await _graphClient.Drives[_containerId].Items[parentFolderId].Children.PostAsync(folder);

Migrate File

// The parameter must be of type Object.
internal async void MigrateFile(Object stateInfo)
{
  var fileStructure = (FileStructure)stateInfo;

  // Check if the file exists in the destination. If it exists
  // - don't upload
  // - check if the file is newer in the source than the destination - then upload
  ...

  // Migrate the file
  // This is where you download the blob as a stream from abs (code below)
  ...

  // Then upload the stream to SPE (code below)
  ...

  // Signal the countdown event that a file has been migrated
  _countdown.Signal();

  return;
}

Downloading From The Blob From ABS As A Stream

BlobClient blobClient = _containerClient.GetBlobClient(blobName);

MemoryStream memoryStream = new MemoryStream();
await blobClient.DownloadToAsync(memoryStream);
memoryStream.Position = 0; // Reset the stream position to the beginning

Uploading The Stream To SPE

int _maxChunkSize = 320 * 1024;

var uploadSessionRequestBody = new CreateUploadSessionPostRequestBody()
{
  AdditionalData = new Dictionary<string, object>
  {
    // Fail is set here, so it doesn't get upload again if it already exist
    { "@microsoft.graph.conflictBehavior", "fail" }
  }
};

var uploadSession = await _graphClient.Drives[_containerId]
    .Items[parentFolderId]
    .ItemWithPath(fileName)
    .CreateUploadSession
    .PostAsync(uploadSessionRequestBody);

// The stream is the same stream from the downloading the blob
var fileUploadTask = new LargeFileUploadTask<DriveItem>(uploadSession, memoryStream, _maxChunkSize, _graphClient.RequestAdapter);
IProgress<long> progress = new Progress<long>(prog => Console.WriteLine($"Uploaded {fileName} {prog} bytes"));

// Check uploadResult.UploadSucceeded to see if it is successful
var uploadResult = await fileUploadTask.UploadAsync(progress);

Overview Of The Sample App

Description

A sample app called MigrateABStoSPE that is designed to migrate files from an Azure Blob Storage (ABS) container to a SharePoint Embedded (SPE) container. The code snippets provided in the Migrating Data from Azure Blob Storage container to SharePoint Embedded container are from the sample app.

It uses Azure.Storage.Blobs and Newtonsoft.Json libraries for working with ABS and JSON data respectively. The app authenticates with both ABS and SPE using client credentials and performs the migration of files.

Packages

Microsoft Graph SDK (version 5.56.0)
Azure.Identity (version 1.12.0)
Azure.Storage.Blobs (version 12.21.0)
CommandLineParser (version 2.9.1)
Newtonsoft.Json (13.0.3)

Out Of Scope

How to deal with files that already exist in the destination - it fails, it doesn't overwrite or rename
How to deal with ABS version newer than the destination - it fails because the file already exists in the destination

Running The Sample App

Open a terminal or command prompt.
Navigate to the directory where the Program.cs file is located.
Make sure you have the .NET Core SDK installed on your machine. You can check this by running the command dotnet --version in the terminal. If the command isn't recognized, you can download and install the .NET Core SDK from the official Microsoft website.
Once you have confirmed that the .NET Core SDK is installed, you can build the application by running the command dotnet build. This will compile the code and generate the necessary binaries.
After the build process is complete, you can run the application by executing the command dotnet run followed by the required arguments. The required arguments are:
- The container-level SAS URL: This is an Azure Blob container level SAS URL. It provides access to the container and its blobs.
- The SPE tenant ID: This is the tenant you're authenticating against in the SPE.
- The SPE client ID: This is the client you're authenticating against in the SPE.
- The SPE container ID: This is the container you're migrating content to in the SPE. For more information on how to get the container ID
- (optional) File name with full path that contains the blob list.
- (optional) File name with full path where to output failed blobs.

For example, the command to run the application with the required arguments would look like this:

dotnet run Program.cs -- --sasurl "<sas url>" --tenantid "<tenant id>" --clientid "<client id>" --containerid "<container id>" [ --blobfile "<file name>" --outputfile "<file name>" ]

Blob and SPE Item Structure

ABS container doesn't adhere to a folder structure, all the blobs are stored in a flat listing structure. When migrating to SPE, the sample app parses the blob name and creates the folder structure in the container ID provided, with the container name as the top folder. If you're migrating to the root folder, you can ignore this section.

Source

Container Name: Container1
- Blob name: FolderA/blob1.txt
- Blob name: FolderA/FolderB/blob2.txt
- Blob name: FolderA/FolderB/FolderC/blob3.txt

Destination

Drive Item folder
- Container1
  - FolderA
    - blob1.txt
    - FolderB
      - blob2.txt
      - FolderC
        
        blob3.txt

Handling Errors and Exceptions

Common Issues

File already exists in the destination
- This app checks to see if the file name exists in the destination before it uploads. If there's a file with the exact same name, it will not do the upload again. It will print to stdout a message that the file already exists. To fix it, you can either delete the file from the destination or change the conflictBehavior to replace and not call CheckIfItemExists on upload.
The file for the list of blobs isn't found
The format for the list of blobs - one blob per line, without quotes around the blob name
Not giving enough permission to access the ABS container
- The minimum permissions are Read and List
Not giving enough permissions to the SPE container
- The required scope is "User.Read" and "FileStorageContainer.Selected"
- Remember to grant admin consent
- Remember to create the mobile & console platform app

Testing the Migration

Verification

When the file is queued, it will print to stdout
It will print the stats of the total blobs that were processed: total, success, exists in destination, and failed.
If there are errors, it will send the failed blob list to a file. The file name will be printed to stdout. It will also, print a command for an incremental re-run.

Conclusion

Summary

In this tutorial, we explored how to migrate content from ABS container to SPE container. By following the steps outlined, writing your own app to migrate content should be easy.

To recap, we:

Authenticated with ABS and Graph
How to use a thread pool to queue migration of blob
Check if the item exists in destination
Retrieved the blob list from ABS container
Uploaded the blob to the SPE container

Understanding these steps is crucial for migrating content from ABS container to SPE container. Now, try implementing these steps in your own projects and see the difference it makes!

Happy coding!

Next Steps

For more information about Blob Storage, see Blob Storage documentation.
For more information about SPE, see SharePoint Embedded documentation.

Appendix

Code Repository

The sample app can be found in the SharePoint Embedded Samples repository.

Last updated on 2025-03-05

Share via

Tutorial For Migrating Content From Azure Blob Storage Container To SharePoint Embedded Container

Purpose

Prerequisites

Authentication

Azure Blob Storage

SharePoint Embedded

Migrating Data from Azure Blob Storage container to SharePoint Embedded container

Description

Connecting to Azure Blob Storage Container

Connecting to SharePoint Embedded

Getting the blob list

Thread pooling

FileStructure

Traverse blob list

Traverse blob name

Check if the item exists

Create folder

Migrate File

Downloading From The Blob From ABS As A Stream

Uploading The Stream To SPE

Overview Of The Sample App

Description

Packages

Out Of Scope

Running The Sample App

Blob and SPE Item Structure

Handling Errors and Exceptions

Common Issues

Testing the Migration

Verification

Conclusion

Summary

Next Steps

Appendix

Code Repository

Additional resources