Migrate 10 TB data to cloud / archive solution that can easily be accessed and at any point in time

Troops 11 Reputation points
2023-12-25T14:05:43.64+00:00

Hi all,

I really hope that someone can help us with our requirement. I have spent couple of days trying to find what is the best solution but still don't know what do use.

Requirement: We are in process of decommissioning our on-prem environment and one of the biggest issues is our file server. We have ca 10 TB data, big CAD files and that kind of files (drawing). We need to find solution in the cloud which will be like archive (we need to save those files at least 15 years):

  1. We need to be able to access files directly at any time
  2. Permissions needs to be configured on the site so that only few people can download / access it
  3. Need to be able to easily access files

I checked azure archive tier but that is not an option because it is a offline storage. I think that no even hot/cold tier is right here because it is not easy to access files in storage account for random user.

Sharepoint online might be an option. But is there anything else what we can use besides Sharepoint Online or Azure storage account for this case? Anyone else had a similar issue / requrement? Microsoft should have solution for this.

Please help.

Microsoft 365
Microsoft 365
Formerly Office 365, is a line of subscription services offered by Microsoft which adds to and includes the Microsoft Office product line.
5,727 questions
Azure Storage Accounts
Azure Storage Accounts
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
3,366 questions
SharePoint
SharePoint
A group of Microsoft Products and technologies used for sharing and managing content, knowledge, and applications.
11,193 questions
0 comments No comments
{count} votes

1 answer

Sort by: Most helpful
  1. Sumarigo-MSFT 47,486 Reputation points Microsoft Employee
    2023-12-28T09:00:53.7366667+00:00

    @Troops Welcome to Microsoft Q&A Forum, Thank you for posting your query here

    Based on your requirements, I would recommend using Azure Blob Storage and Azure Files Storage with Azure AD authentication and Azure Blob Storage lifecycle management.

    Azure Blob Storage is a scalable and cost-effective solution for storing unstructured data such as files, images, and videos. You can store up to 5 PB of data in a single storage account, and you can access the data directly from anywhere in the world using HTTP or HTTPS.

    Azure AD authentication allows you to control access to your data by requiring users to authenticate with their Azure AD credentials before they can access the data. You can also configure role-based access control (RBAC) to grant different levels of access to different users.

    Azure Blob Storage lifecycle management allows you to automatically move data between different storage tiers based on its age and access patterns. For example, you can configure lifecycle management to move data to the archive tier after it has been in the cool tier for a certain amount of time. This can help you reduce costs by storing data in the most cost-effective tier for its current usage pattern.

    To meet your specific requirements, I would recommend the following configuration:

    1. Create an Azure Blob Storage account with the hot tier enabled.
    2. Configure Azure AD authentication for the storage account.
    3. Create a container in the storage account for each set of files that requires different permissions.
    4. Configure RBAC to grant access to the appropriate users for each container.
    5. Configure lifecycle management to move data to the cool tier after a certain amount of time, and to the archive tier after 15 years.

    With this configuration, you can access the files directly from anywhere in the world using HTTP or HTTPS, and you can control access to the files using Azure AD authentication and RBAC. You can also reduce costs by automatically moving data to the most cost-effective tier based on its age and access patterns.

    Authorize access to blobs using Microsoft Entra ID

    Azure Storage migration overview

    Azure Files Storage with AAD authentication: Azure file shares are suitable for general-purpose file data. This data includes anything you use an on-premises SMB share for. With Azure File Sync, you can cache the contents of several Azure file shares on servers running Windows Server on-premises.

    Overview of Azure Files identity-based authentication options for SMB access

    Access Azure file shares using Microsoft Entra ID with Azure Files OAuth over REST

    There are various migration tools that you can use to perform the migration. Some are open source like AzCopy, robocopy, xcopy, and rsync while others are commercial. List of available commercial tools and comparison between them is available on our comparison matrix.

    Open-source tools are well suited for small-scale migrations. For migration from Windows file servers to Azure Files, Microsoft recommends starting with Azure Files native capability and using Azure File Sync. For more complex migrations consisting of different sources, large capacity, or special requirements like throttling or detailed reporting with audit capabilities, commercial tools are the best choice. These tools make the migration easier and reduce the risk significantly. Most commercial tools can also perform the discovery, which provides a valuable input for the assessment.

    Please let us know if you have any further queries. I’m happy to assist you further.


    Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.


Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.