Migrating from AWS RDS Postgresql to Azure for Postgres flexible server - Big table sizes

LuisC 20

The issue I have is that our tables have big sizes, for example, 500Gb, and I saw in here that we have a limit of 1TB. My issue would be solved if I could use 2 DMS instances pointing to the same database but different tables, is that even possible?

Accepted answer

Oury Ba-MSFT 19,506 Reputation points Microsoft Employee

2024-07-18T18:51:29.6833333+00:00

@LuisC Thank you for reaching out.

In addition to the above answer provided by akinbade abiola.

The above link you are looking at is our old data migration service which is replaced by our new service Migration service in Azure Database for PostgreSQL - Azure Database for PostgreSQL - Flexible Server | Microsoft Learn.

We have updated the documentation, but some areas are currently being updated including the routing from old to new service. Our new service not only have “No Size Limitations” but it also parallelizes large table into multiple batches to speed up the performance. It is executed directly from the Flexible Server.

Regards,

Oury
Please sign in to rate this answer.
LuisC 20 Reputation points

2024-07-18T20:04:23.12+00:00

Thanks for your answer, the issue I have with that service is that the DB in AWS is in a VPC with no public access, and in Azure our DB is exposed through a private endpoint. In here we have the following limitation:

The migration service is unable to perform migration when the source database is Azure Database for PostgreSQL - Single Server with no public access or is an on-premises/AWS using a private IP, and the target Azure Database for PostgreSQL - Flexible Server instance is accessible only through a private endpoint.

So we can't go this way.

Oury Ba-MSFT 19,506 Reputation points Microsoft Employee

2024-07-22T21:34:22.5633333+00:00

LuisC Thanks a lot for pointing it out. We will remove the limitation on immediate basis. To support this scenario, we have introduced Migration runtime server feature in the service.

https://learn.microsoft.com/en-us/azure/postgresql/migrate/migration-service/concepts-migration-service-runtime-server

Regards,

Oury

LuisC 20 Reputation points

2024-07-22T21:45:33.6766667+00:00

Thanks for letting me know about this! I'll try it out.

Oury Ba-MSFT 19,506 Reputation points Microsoft Employee

2024-07-23T16:48:53.2733333+00:00

Thank you for accepting the answer. I am glad it was helpful.

Please see my new comment also.

Regards,

Oury
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

1 additional answer

akinbade abiola 18,300 Reputation points

2024-07-17T22:43:43.2266667+00:00

Using multiple DMS instances for different tables in the same database is not supported.

You can attempt a phased migration by migrating first the smaller tables and then implement horizontal partitioning for your large tables

You can mark it 'Accept Answer' and 'Upvote' if this helped you

Regards,

Abiola
Please sign in to rate this answer.
LuisC 20 Reputation points

2024-07-18T12:11:00.0966667+00:00

How would a phased migration work with DMS?

Oury Ba-MSFT 19,506 Reputation points Microsoft Employee

2024-07-23T16:48:06.3+00:00

LuisC

Migration runtime server is recently announced in public preview. With respect to custom DNS, are you using custom DNS in the target Flexible server?

There are two specific limitations specified in the documentation –

The migration service is unable to perform migration when the source database is Azure Database for PostgreSQL - Single Server with no public access or is an on-premises/AWS using a private IP, and the target Azure Database for PostgreSQL - Flexible Server instance is accessible only through a private endpoint.

This point will be removed from the documentation as “Migration runtime server” feature was introduced to handle this scenario.

The Migration Runtime Server is designed to operate with the default DNS servers/private DNS zones, for example, privatelink.postgres.database.azure.com. Custom DNS names/DNS servers aren't supported by the migration service when you use the Migration Runtime Server feature. When you're configuring private endpoints for both the source and target databases, it's imperative to use the default private DNS zone provided by Azure for the private link service. The use of custom DNS configurations isn't yet supported and might lead to connectivity issues during the migration process.

As of now, migration runtime server works with the default DNS private zones. In your scenario,

Is the target using custom DNS zone?

Also, we provide GA level support for public preview feature as that is mostly the concern of the customers for using preview features.

LuisC 20 Reputation points

2024-07-23T16:56:30.8533333+00:00

Yes, we are using custom DNS servers. Is there a workaround for this?

Oury Ba-MSFT 19,506 Reputation points Microsoft Employee

2024-07-24T15:27:46.8466667+00:00

LuisC

Migration runtime server residing between source and target should have default DNS zone. Can we sync offline to discuss further on this.

I will send you a private message.

Regards,

Oury
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Migrating from AWS RDS Postgresql to Azure for Postgres flexible server - Big table sizes

1 additional answer

Your answer