Apologies for the delayed feedback on this issue. Here is what was happening to our farm:
I wanted to share this info hoping it will help others. The main issue around SP performance once we went live was related to Microsoft SQL server. The Production SQL servers were all configured with multiple sockets – each socket having 1 core. The server was allocated 16 sockets – each with 1 cores to meet the 16 cores requested for this server. The problem is that the Standard Edition of SQL server can only address 4 sockets. Since each socket had 1 core, SQL could only use 4 cores total. This was not a problem when we had a few dozen people on the new sites for testing. However, as soon as we went live, now we had a few thousand users and very soon we started to see the impact on SharePoint.
The lesson here is that, any product that is licensed based on CPU should be taken into consideration when it comes to VM configuration. In this case, for SQL, we should not surpass 4 sockets for any SQL VM that will be running the Standard SQL version. For other products, there may be different licensing requirements which should be taken into consideration. In our case, to achieve 16 cores, the VM was reconfigured from 16 sockets to 4 sockets - each socket having 4 cores.
Below is some screenshots that will visualize this issue.
(the screenshots are for a VM with 8 cpu cores but the idea is the same)