The issue you are facing is due to the fact that Excel has a 55-second timeout when fetching data from SharePoint, and it is not related to the calculation timeout setting. Therefore, increasing the calculation timeout setting will not solve the problem. The timeout value can be increased by using a registry key, as described in the Microsoft article "Unable to export to Excel from SharePoint". However, this will only affect the time it takes to fetch data from SharePoint, not the calculation timeout.
I don't know of any way to increase the calculation timeout beyond the maximum of 1,800 seconds (30 minutes) in Excel. It is possible that Excel is interpreting any value greater than 1,800 seconds as 30 seconds due to a limitation in the software.
I suggest breaking up your calculation into smaller parts that take less than 1,800 seconds each, or finding a way to optimize your calculation to reduce the overall time it takes to complete.
References: