Pipelines that require a scale up of the db are not scaling up

Question

Pipelines that require a scale up of the db are not scaling up

Diana Rus 55

Hi,

I have a couple of pipelines that have a stored procedure activity that scale up a database before the actual processing of the data starts. This has been working fine for almost a year, however in the past 2 days, the scale up times out and subsequently the pipelines fail. Has something changed or is there a better way to get a db to scale up automatically when needed?

Thank you!

phemanth 15,755 Reputation points Microsoft External Staff Moderator

2024-06-25T09:12:02.4466667+00:00
@Diana Rus

Thanks for using MS Q&A platform and posting your query.

There could be several reasons why your scale-up operations are timing out. Here are a few possibilities and potential solutions:

Increased Load or Changes in Database Usage: If your database has seen a significant increase in load or changes in usage patterns, this could potentially cause scale-up operations to take longer than usual.

Monitor Scale Operations: You can monitor the progress of the scale operation using the following query: SQL
SELECT * FROM sys.dm_operation_status WHERE major_resource_id = ‘YourDb’ ORDER BY start_time DESC;
If you see a long time has elapsed since the scale process started then you can trigger a PowerShell script.

Database Size: The physical size of the database and the number of records can impact the performance of scale operations. If your database has grown significantly, this might be contributing to the issue.

Auto-scaling Solutions: There are several auto-scaling solutions available that can help manage database scaling more efficiently. For instance, Amazon offers DynamoDB auto scaling and AWS Auto Scaling, and Azure provides automation runbooks or Azure Logic apps for automatic scaling. These services can adjust capacity in real-time as demands change, helping to maintain optimal application performance and availability.

Database Scaling Techniques: Other techniques such as caching database queries, using database indexes, data replication, and sharding can also help improve the performance of your database

Remember, it’s important to regularly monitor your database and adjust your scaling strategies as needed to ensure optimal performance

Hope this helps. Do let us know if you any further queries.
phemanth 15,755 Reputation points Microsoft External Staff Moderator

2024-06-27T10:39:33.6433333+00:00

@Diana Rus We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Diana Rus 55 Reputation points

2024-06-27T10:41:49.41+00:00

Hi, no resolution and it seems to be a very sporadic event, so far this week, it's been 50/50 if it scales up and down
phemanth 15,755 Reputation points Microsoft External Staff Moderator

2024-07-02T16:22:41.43+00:00
@Diana Rus Here are some potential solutions:

Monitor Scale Operations: You can monitor the progress of the scale operation using the following SQL query:

SELECT * FROM sys.dm_operation_status WHERE major_resource_id = 'YourDb' ORDER BY start_time DESC;

If a long time has elapsed since the scale process started, you can trigger a PowerShell script.

Database Size: The physical size of the database and the number of records can impact the performance of scale operations. If your database has grown significantly, this might be contributing to the issue.

Auto-scaling Solutions: There are several auto-scaling solutions available that can help manage database scaling more efficiently. For instance Azure provides automation runbooks or Azure Logic apps for automatic scaling. These services can adjust capacity in real-time as demands change, helping to maintain optimal application performance and availability.

Database Scaling Techniques: Other techniques such as caching database queries, using database indexes, data replication, and sharding can also help improve the performance of your database.Here are some potential solutions
phemanth 15,755 Reputation points Microsoft External Staff Moderator

2024-07-03T10:05:42.3866667+00:00

@Diana Rus just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Your answer

phemanth 15,755 Reputation points Microsoft External Staff Moderator

2024-06-25T09:12:02.4466667+00:00

@Diana Rus

Thanks for using MS Q&A platform and posting your query.

There could be several reasons why your scale-up operations are timing out. Here are a few possibilities and potential solutions:

Increased Load or Changes in Database Usage: If your database has seen a significant increase in load or changes in usage patterns, this could potentially cause scale-up operations to take longer than usual.

Monitor Scale Operations: You can monitor the progress of the scale operation using the following query: SQL
SELECT * FROM sys.dm_operation_status WHERE major_resource_id = ‘YourDb’ ORDER BY start_time DESC;
If you see a long time has elapsed since the scale process started then you can trigger a PowerShell script.

Database Size: The physical size of the database and the number of records can impact the performance of scale operations. If your database has grown significantly, this might be contributing to the issue.

Auto-scaling Solutions: There are several auto-scaling solutions available that can help manage database scaling more efficiently. For instance, Amazon offers DynamoDB auto scaling and AWS Auto Scaling, and Azure provides automation runbooks or Azure Logic apps for automatic scaling. These services can adjust capacity in real-time as demands change, helping to maintain optimal application performance and availability.

Database Scaling Techniques: Other techniques such as caching database queries, using database indexes, data replication, and sharding can also help improve the performance of your database

Remember, it’s important to regularly monitor your database and adjust your scaling strategies as needed to ensure optimal performance

Hope this helps. Do let us know if you any further queries.
phemanth 15,755 Reputation points Microsoft External Staff Moderator

2024-06-27T10:39:33.6433333+00:00

@Diana Rus We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Diana Rus 55 Reputation points

2024-06-27T10:41:49.41+00:00

Hi, no resolution and it seems to be a very sporadic event, so far this week, it's been 50/50 if it scales up and down
phemanth 15,755 Reputation points Microsoft External Staff Moderator

2024-07-02T16:22:41.43+00:00

@Diana Rus Here are some potential solutions:

Monitor Scale Operations: You can monitor the progress of the scale operation using the following SQL query:

SELECT * FROM sys.dm_operation_status WHERE major_resource_id = 'YourDb' ORDER BY start_time DESC;

If a long time has elapsed since the scale process started, you can trigger a PowerShell script.

Database Size: The physical size of the database and the number of records can impact the performance of scale operations. If your database has grown significantly, this might be contributing to the issue.

Auto-scaling Solutions: There are several auto-scaling solutions available that can help manage database scaling more efficiently. For instance Azure provides automation runbooks or Azure Logic apps for automatic scaling. These services can adjust capacity in real-time as demands change, helping to maintain optimal application performance and availability.

Database Scaling Techniques: Other techniques such as caching database queries, using database indexes, data replication, and sharding can also help improve the performance of your database.Here are some potential solutions
phemanth 15,755 Reputation points Microsoft External Staff Moderator

2024-07-03T10:05:42.3866667+00:00

@Diana Rus just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Share via

Pipelines that require a scale up of the db are not scaling up

Your answer