Issue with IIS 10 - 502.3 error with win-sc32-status 1236 when having high load of request

Question

Issue with IIS 10 - 502.3 error with win-sc32-status 1236 when having high load of request

Minh Nhật 1

I'm having a lot of 502.3 error when my production website under high load of requests, win-sc32-status mostly is 1236 (and I find that it's difficult to understand what can cause the problem). And I've created a test environment in order to reproduce the 502.3 error (mimic high load of requests using JMeter) but the status I received from win-sc32-status mostly is 64 and 12002
So my question is:

how can I reproduce the 502.3 error with sc-win32-status 1236
what is the best describe of sc-win32-status 1236 - The network connection was aborted by the local system is kind of confuse to me (what is the local system here?, how and why it is aborted?)
a real life scenario where sc-win32-status 1236 can happen

Bruce Zhang-MSFT 3,776 Reputation points

2022-06-06T06:43:45.707+00:00

Hi @Minh Nhật ,

502.3 means timeout and status 1236 means network connection was aborted. I think it is very likely that this problem is caused by session loss or expiration. Requests cannot be authenticated when the session is lost, and your production environment will be under high load. This may cause the request to time out. When the data in the request is lost, especially when the session is stored in the server, it will make the server feel that the connection is not secure. Maybe it's because of this that status 1236.
Minh Nhật 1 Reputation point

2022-06-06T07:18:32.997+00:00

thank you @Bruce Zhang-MSFT ,
yeah that makes sense, right now there's an IIS recycling policy setup daily in production environment and it's using InProc session as well, so I guess all of the session from users are loss after the recycle process. I will try that case and update into this thread
Minh Nhật 1 Reputation point

2022-06-06T08:09:11.067+00:00

@Bruce Zhang-MSFT just notice that there's a sc-win32-status code 121 in production environment as well, in which I cannot reproduce in testing environment. Is that something related to the connectivity between clients and my server?
Bruce Zhang-MSFT 3,776 Reputation points

2022-06-07T02:21:50.947+00:00

Hi @Minh Nhật ,

Essentially 121 also means connection timeout. So no matter 121 or 1236, it is a status code returned due to connection interruption or timeout.
Minh Nhật 1 Reputation point

2022-06-07T07:19:34.32+00:00

I've created a scenario that having IIS application pool recycled when it's under JMeter testing. Having 502.3 this time but with sc-win32-status code of 12002 this time. There's one thing that the testing server, I created JMeter requests call into it directly, but with the production environment, all of the requests must go through an ELB, could this be the reason why there's a difference of sc-win32-status code between testing and production environment?
Bruce Zhang-MSFT 3,776 Reputation points

2022-06-08T01:37:06.64+00:00

Hi @Minh Nhật ,

It's quite possible. Have you tested in production to cancel the ELB and send the request directly to the content server?
Minh Nhật 1 Reputation point

2022-06-10T01:48:28.537+00:00

Hi @Bruce Zhang-MSFT
Sadly that I cannot interfere with the production like that due to customer compliance, having the addition ELB in testing env will take time and cost as well.
However, if I change the timeout of IIS server to higher number, number of issues are reduced, so can we come with a conclusion that the server cannot handle that much of requests and it's better for customer to scale up their services? (we're using cloud, but the application is built using a platform that will have license cost if we want to scale up/out, so we just want to make sure that there's no misconfiguration from IIS)
Bruce Zhang-MSFT 3,776 Reputation points

2022-06-10T07:40:15.263+00:00

Hi @Minh Nhật ,

if I change the timeout of IIS server to higher number, number of issues are reduced,

According to your test result, I think scale up services is a better choice. This can effectively improve the overall server's ability to handle a large number of requests and avoid excessive request queuing. But you can also optimize the speed and queue size of your application processing requests through code. If so, the Q&A community is not a place to offer practical advice. I think the better place is creating a support ticket.
Minh Nhật 1 Reputation point

2022-06-13T10:27:33.583+00:00

thank you so much for your help, we're trying different approach to improve the performance now, really appreciate it!

Your answer

Bruce Zhang-MSFT 3,776 Reputation points

2022-06-06T06:43:45.707+00:00

Hi @Minh Nhật ,

502.3 means timeout and status 1236 means network connection was aborted. I think it is very likely that this problem is caused by session loss or expiration. Requests cannot be authenticated when the session is lost, and your production environment will be under high load. This may cause the request to time out. When the data in the request is lost, especially when the session is stored in the server, it will make the server feel that the connection is not secure. Maybe it's because of this that status 1236.
Minh Nhật 1 Reputation point

2022-06-06T07:18:32.997+00:00

thank you @Bruce Zhang-MSFT ,
yeah that makes sense, right now there's an IIS recycling policy setup daily in production environment and it's using InProc session as well, so I guess all of the session from users are loss after the recycle process. I will try that case and update into this thread
Minh Nhật 1 Reputation point

2022-06-06T08:09:11.067+00:00

@Bruce Zhang-MSFT just notice that there's a sc-win32-status code 121 in production environment as well, in which I cannot reproduce in testing environment. Is that something related to the connectivity between clients and my server?
Bruce Zhang-MSFT 3,776 Reputation points

2022-06-07T02:21:50.947+00:00

Hi @Minh Nhật ,

Essentially 121 also means connection timeout. So no matter 121 or 1236, it is a status code returned due to connection interruption or timeout.
Minh Nhật 1 Reputation point

2022-06-07T07:19:34.32+00:00

I've created a scenario that having IIS application pool recycled when it's under JMeter testing. Having 502.3 this time but with sc-win32-status code of 12002 this time. There's one thing that the testing server, I created JMeter requests call into it directly, but with the production environment, all of the requests must go through an ELB, could this be the reason why there's a difference of sc-win32-status code between testing and production environment?
Bruce Zhang-MSFT 3,776 Reputation points

2022-06-08T01:37:06.64+00:00

Hi @Minh Nhật ,

It's quite possible. Have you tested in production to cancel the ELB and send the request directly to the content server?
Minh Nhật 1 Reputation point

2022-06-10T01:48:28.537+00:00

Hi @Bruce Zhang-MSFT
Sadly that I cannot interfere with the production like that due to customer compliance, having the addition ELB in testing env will take time and cost as well.
However, if I change the timeout of IIS server to higher number, number of issues are reduced, so can we come with a conclusion that the server cannot handle that much of requests and it's better for customer to scale up their services? (we're using cloud, but the application is built using a platform that will have license cost if we want to scale up/out, so we just want to make sure that there's no misconfiguration from IIS)
Bruce Zhang-MSFT 3,776 Reputation points

2022-06-10T07:40:15.263+00:00

Hi @Minh Nhật ,

if I change the timeout of IIS server to higher number, number of issues are reduced,

According to your test result, I think scale up services is a better choice. This can effectively improve the overall server's ability to handle a large number of requests and avoid excessive request queuing. But you can also optimize the speed and queue size of your application processing requests through code. If so, the Q&A community is not a place to offer practical advice. I think the better place is creating a support ticket.
Minh Nhật 1 Reputation point

2022-06-13T10:27:33.583+00:00

thank you so much for your help, we're trying different approach to improve the performance now, really appreciate it!

Share via

Issue with IIS 10 - 502.3 error with win-sc32-status 1236 when having high load of request

Your answer