Hi, I'm designing a disaster recovery solution but I have some doubts, in particular I'm not sure about: What happens when a region fails back? Are the resources I had in that region lost and will have to be recreated or will they "recover" on their own? For example, does AKS restart everything automatically? With respect to the last question, assuming that AKS manages to failback, if I had a storage account shared between AKS of the primary region and AKS of the secondary region (DR), how can I avoid them writing to the storage account at the same time?

That depends on the service - for example, it's handled differently for Azure Storage, Azure SQL, and Azure VMs. In general (with some exceptions), both failover and failback between the regions would require some involvement on your part because of the potential for data loss. For example, Azure Site Recovery allows you to automate orchestration of failover and failback, but you need to initiate each. AKS does not support cross-region failover/failback - you need to implement either active/passive or active/active configuration. Details at https://learn.microsoft.com/en-us/azure/aks/ha-dr-overview hth Marcin

Disaster recovery scenario

Nunzio Mauro 0

Hi,

I'm designing a disaster recovery solution but I have some doubts, in particular I'm not sure about:

What happens when a region fails back? Are the resources I had in that region lost and will have to be recreated or will they "recover" on their own? For example, does AKS restart everything automatically?
With respect to the last question, assuming that AKS manages to failback, if I had a storage account shared between AKS of the primary region and AKS of the secondary region (DR), how can I avoid them writing to the storage account at the same time?

1 answer

Marcin Policht 24,530 Reputation points MVP

2024-03-25T21:26:40.4933333+00:00
That depends on the service - for example, it's handled differently for Azure Storage, Azure SQL, and Azure VMs. In general (with some exceptions), both failover and failback between the regions would require some involvement on your part because of the potential for data loss. For example, Azure Site Recovery allows you to automate orchestration of failover and failback, but you need to initiate each.

AKS does not support cross-region failover/failback - you need to implement either active/passive or active/active configuration. Details at https://learn.microsoft.com/en-us/azure/aks/ha-dr-overview

hth

Marcin
Please sign in to rate this answer.
Nunzio Mauro 0 Reputation points

2024-03-25T22:05:51.6666667+00:00

First of all, thanks for the answer, maybe I expressed myself badly, what I meant in the second question, in the event that a region goes down and then goes back up, how does AKS behave? Does the service instance disappear or is it just there and starts working again?
I ask all this because my concern is the storage account that the two instances have in common could be at a time where the two AKS instances are active and writing concurrently and I would like to avoid this.

I had thought of using private endpoints and creating or approving them only for the AKS instance that I want active at that moment, but the problem is that if the region goes down it is no longer possible to make changes from what I read here:

If a resource group's region is temporarily unavailable, you may not be able to update resources in the resource group because the metadata is unavailable. The resources in other regions still function as expected, but you may not be able to update them. This condition may also apply to global resources like Azure DNS, Azure DNS Private Zones, Azure Traffic Manager, and Azure Front Door. You can view which types have their metadata managed by Azure Resource Manager via the list of types for the Azure Resource Graph resources table.

Marcin Policht 24,530 Reputation points MVP

2024-03-25T22:37:39.57+00:00

That's impossible to tell - ultimately it depends on the reason behind the failure of the primary region. Microsoft will attempt to restore services to their state before the failure, but to what extent this can be done is obviously not something anyone can predict.

Effectively, you should not based your DR strategy on any assumptions about the recoverability of the primary region.

Instead, make sure that the AKS cluster in the secondary region will be able to take over once the primary is down.

hth

Marcin

Nunzio Mauro 0 Reputation points

2024-04-03T19:14:33.11+00:00

OK clear, and precisely because we don't know the behavior by putting myself in the worst case I would like to avoid any writes to the storage account from AKS in PRD while the failover is active. Writes that could be automatically generated by cronjobs for example.

Marcin Policht 24,530 Reputation points MVP

2024-04-03T21:08:23.2666667+00:00

Yep - you got it...

If the above response helps answer your question, remember to "Accept Answer" so that others in the community facing similar issues can easily find the solution. Your contribution is highly appreciated.

hth

Marcin

Nunzio Mauro 0 Reputation points

2024-04-04T06:05:32.37+00:00

Do you have any suggestions for avoiding these writings?

Marcin Policht 24,530 Reputation points MVP

2024-04-04T22:46:45.6533333+00:00

Which writings?

Nunzio Mauro 0 Reputation points

2024-04-05T01:16:13.8566667+00:00

I'll try to better describe my disaster recovery scenario, I have two AKS instances that access a geo-redundant storage account via private endpoint. In case everything works, only AKS in the primary region has cronjobs that write to the storage account (also deployed in the primary region) while AKS in the secondary region is on standby. When the primary region is down, failover to the storage account is active and AKS is also active in the secondary region which will start cronjobs that will write to the storage account. Now in the worst case scenario in which AKS of the primary region were to come back up it would also restart its cronjobs which would write concurrently together with the cronjobs of AKS of the secondary region on the storage account. How can I avoid this possible scenario? Is there a way to avoid blocking writes from the primary region's AKS cronjobs since I can't remove the private endpoint from the geo-redundant storage account while the region it is deployed in is down?
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Disaster recovery scenario

1 answer

Your answer