Azure SQL Server Database over Large Data Transactions

Question

I've hosted a few Azure Functions which interact with Azure SQL Server Database once per day to Read the Data, Perform Calculations, and Save Calculated Data back to SQL Server Database.

Every day, Azure Function reads around 84 million records and performs the calculation over that data. Once the calculation is done, it saves that data back to the Database again.

Considering the above scenario, I have got a few concerns. Please advice:

1. Is Azure SQL Server Database worth/suitable for such a Volume of data?
2. If not, then what'd be the better Alternative?
3. Is there any better approach to handle this situation?

Answer 1

Hi @Shreekumar Suggamad Thank you for posting your question on Microsoft Q&A and for using Azure services.

1. I agree with @Alberto Morillo , Azure SQL database is suitable for your scenario and can handle customer data up to 2 trillion of row in a single table.
2. No other alternative than using Azure SQL
3. Yes, moving data in or out any database is expensive, so if you can push calculation to the database would be much better. What kind of calculation you need to do? Can they be done using T-SQL only or you need a more complex language, like Python or C# as you need to do a lot of checks and controls (SQL is a computationally complete language, so you should be able to do anything you need for what concerns calculations)

Hope that helps

Regards,
Oury

Answer 2

1. To see the high scale and performance of Azure SQL Database at its best, to see Azure SQL with the ability to insert 1.4 million rows per second incorporate a non-durable memory-optimized table to speed up data ingestion. You can offload later historical data from the In-Memory table to a disk based Columnstore table.
2. You can do it with Azure SQL Database.
3. With the adjustments on answer #1 you should be fine