PostgreSQL Flex - storing Terrabytes of data and setting up scheduled pipelines

Question

1. is it ok to store and manage Terabytes of data?

2.Azure functions or Azure Data factory and Azure Databricks which one is best business practice to use to schedule pipelines.

Answer 1

Hi, @Grover, Jyoti Welcome to Microsoft Q&A thanks for posting your question.

Azure PostgreSQL Flexible Server is a managed service offering that is designed to provide a flexible and scalable The service is engineered with specific limits for compute and storage resources, which are listed in the documentation you provided here.

https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/concepts-compute-storage

As long as your workload stays within these limits, there is no reason why you should not use Azure PostgreSQL Flexible Server. The service provides a number of benefits, including automatic backups, high availability, and scalability, which can help you to manage your PostgreSQL workloads more effectively and efficiently.

On the second question the choice between these services depends on your specific requirements and use case. to run small, event-driven tasks, Azure Functions may be the best choice. for complex data pipelines with multiple sources and destinations, Azure Data Factory may be a better fit. If you need to process large amounts of data and build machine learning models, Azure Databricks may be the best choice.

Regards

Geetha