Not sure if this is the write forum to ask this question. please direct me to the proper place if not. We have a .NET 8 C# function app (V4), running in isolation mode. It is a gRPC client connecting to a gRPC server also running in Azure, under a dedicated app service plan, processing some relatively high volume transactions. Over the past few weeks we've seen a spike in errors with the following error message - "Bad gRPC response. HTTP status code: 500". If I am reading the transaction details correctly in App Insights, the gRPC server completed successfully but the client still generated this 500 response error. Most of the transactions get through, and the ones that don't, our retry process kicks in and they will process successfully the next time. Been searching around for what this error means and some possible causes. Have not really been successful. Any suggestions on things to try. Thanks David

"Bad gRPC response. HTTP status code: 500" error with Azure function app

David Toyer 15

Not sure if this is the write forum to ask this question. please direct me to the proper place if not.

We have a .NET 8 C# function app (V4), running in isolation mode. It is a gRPC client connecting to a gRPC server also running in Azure, under a dedicated app service plan, processing some relatively high volume transactions. Over the past few weeks we've seen a spike in errors with the following error message - "Bad gRPC response. HTTP status code: 500".
User's image

If I am reading the transaction details correctly in App Insights, the gRPC server completed successfully but the client still generated this 500 response error. User's image

Most of the transactions get through, and the ones that don't, our retry process kicks in and they will process successfully the next time. Been searching around for what this error means and some possible causes. Have not really been successful. Any suggestions on things to try.

Thanks
David

Salem Elmrayed 0 Reputation points

2024-06-04T00:15:44.9433333+00:00

We're having the same problem with Azure App Service. On high load, the internal GRPC client which communicates with our identity server throws error 500 without any details, everything on our side meets the best practices, what makes this very bad is that when it happens we must restart the app service to make the error go away.
Adding "ServerGarbageCollection" helped a little bit, but the problem still exists

https://learn.microsoft.com/en-us/aspnet/core/grpc/performance?view=aspnetcore-8.0
dkorolev 5 Reputation points

2024-06-06T08:58:00.94+00:00

This started for us as well not long ago
Salem Elmrayed 0 Reputation points

2024-06-06T10:08:08.68+00:00

We ran tests on Azure Container App no issue there,
Matt 0 Reputation points

2024-06-06T19:46:16.72+00:00

I am having this problem as well. It seems like it has gotten much worst recently.
Peter Macko 0 Reputation points

2024-06-10T12:17:53.4433333+00:00

Hello,

in our case, we have the same observation as you. Web App in Azure, communication using gRPC. from the update to .net8 of our app - 3 weeks ago these errors started to appear. In the peaks, the service starts to generate a lot of gRPC errors. It was worse and worse week by week.

BUT: from the logs, I can see that on 9.6.2024 11:00 +02:00 exactly this problem totally disappeared.

Have you got the same findings?

Any idea what happened? In Sunday there was literally no one who could do something with it.

Peter
David Toyer 15 Reputation points

2024-06-10T15:05:20.04+00:00

In the end, we made the decision to switch back to a REST-based API approach. Was never really able to figure out the source problem. Primary reason we went with gRPC was performance but we can work around this using REST instead. Thanks everyone for you input.

David

Share via

"Bad gRPC response. HTTP status code: 500" error with Azure function app