Storage Account Object replication of subfolders and subfolder contents not working.

Question

Any idea why Object replication of subfolders and their contents in a container do not replicate?

I have some Azure 'Application Insights' Diagnostic Logs being saved in a storage account. The folder structure that Azure uses is quite long. Azure places *.JSON files in subfolder for each hour of the day sometimes down to a subfolder per minute that they occur in. Something I cannot change. For example.

Everything in the container needs to be automatically replicate to another storage account in another customer Tenant. Everything meaning, the container root, the many subfolders, and all the *JSON logs as they are generated each day.

I have set up Object Replication and it works for anything saved in the source container only; but none of the subfolders, and their many subfolders and *.JSON files in those sub folders will replicate.

So yes it works, but not as desired. For example, I uploaded an image manually into the source container and it showed up in the destination account in about 2 minutes. If I manually create a folder, it does not. (I am aware there is no SLA on replication time and it can take a very long time....).

The desire is for everything to be replicated along with subfolders and *.JSON files in them.

I have read everything and all sub documents of https://learn.microsoft.com/en-us/azure/storage/blobs/object-replication-overview#blob-index-tags.

Answer 1

There is no such thing as a "folder" in blob storage. All files are stored flat in the container. This is misleading because most storage UX show you a "folder" structure but that doesn't actually exist. When you name a blob file you can include a "path" in the blob name such as logs/today/data.log. The blob is stored in the container as logs/today/data.log along with all the other blobs. When the UX renders the container it understands that you are using a hierarchical naming convention and it shows you the logs folder but it is just UX magic. All blobs are stored at the "root" of the container, there are no folders.

The point of saying that is that you often will have replication rules in place to select the blobs that you want to copy. If you are looking for certain files then you cannot simply try to match against the base filename (e.g. data.log) as it would not match against blobs that are stored in "folders" since the blob name contains that folder name. Check your replication rules and ensure that you aren't filtering out blobs that are in "folders" because of your rules.

Answer 2

Hello Daniel-4204,

Greetings! Welcome to Microsoft Q&A Platform.

 

Object replication asynchronously copies block blobs between a source storage account and a destination account. To get everything in the container, including all subfolders and their contents, replicates to another storage account in another customer tenant using Azure Object Replication

Below are the troubleshooting steps:

• Once verify object replication policy configuration, make sure that the Object Replication policy is correctly configured at the container level, not only just at the root level. The policy should cover all blobs within the container, including all folders in subfolders.

Azure Storage has a maximum path length of 1,024 characters. If the path length of your subfolders and blobs exceeds this, it might prevent replication. Ensure that your folder structure does not exceed this limit.

•  Use Azure Monitor or Storage Explorer to monitor the replication status. Look for any errors or delays in the replication process. You can also use storage metrics to monitor replication process.
•  If Object Replication continues to fail for subfolders, you can use AzCopy as a backup method to ensure that everything, including subfolders and their contents, is replicated by using "--recursive=true" this will check that all subfolders and their contents are copied.
• Check the destination storage account has the correct permissions to receive and create the nested folder structure and files. If there are permission issues, the replication might fail for specific items.

 

Similar post: https://learn.microsoft.com/en-us/answers/questions/896443/how-long-does-it-take-for-storage-account-object-r

 

Here is the doc for your reference: https://learn.microsoft.com/en-us/azure/storage/blobs/object-replication-configure?tabs=portal

 

Limitations:

• Object replication is supported for general-purpose v2 storage accounts and premium block blob accounts. Both the source and destination accounts must be either general-purpose v2 or premium block blob accounts. Object replication supports block blobs only, "Append blobs and page blobs" are not supported.
• Object replication is supported for accounts that are encrypted with either Microsoft-managed keys or customer-managed keys. It is supported for blobs in the source account that are encrypted with a customer-provided key. https://learn.microsoft.com/en-us/azure/storage/common/customer-managed-keys-overview.

Hope this answer helps! please let us know if you have any further queries. I’m happy to assist you further.

 Please "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.