Best solution to capture all requests/responses of Asp.Net Web API hosted on Azure

Question

My system is an Asp.Net Web API (.Net 5) hosted on Azure kubernetes service. I have a requirement to log all requests/responses for all controllers of the system. Currently, the system logs only exceptions to Application Insights using Middleware and the default logger injected by the framework.

My questions are:

1. Should I develop the new requirement as the current implementation of logging exceptions by adding an additional middleware to log all requests/responses to Application Insights?
2. Are there any services/configurations on Azure to log all requests/responses so that I don't need to modify the code?
3. What are the best practices for that kind of requirement? Is this ok to save all requests/responses into Application Insights to easily query/trace/investigate issues later or it should be somewhere else such as databases. etc. since the requests/responses may have sensitive info?

Answer 1

Hi @Chau Tieu

Welcome to Microsoft Q&A! Apologies for the delayed response.

Application Insights is an extensible Application Performance Management (APM) service. This is for mainly instrumentation (measure of performance) purpose. Here is a list of all the scenarios monitored by Application Insights: What does Application Insights Monitor?.
Generally, the record of **all* request and response are monitored from client side for diagnostics and troubleshooting purpose as mentioned in this article: HTTP trace from client.

In view of the points above, please find below my response to your queries:

1. Should I develop the new requirement as the current implementation of logging exceptions by adding an additional middleware to log all requests/responses to Application Insights?
It depends on your requirement. If it is absolutely required to record all the requests and responses on the server side, you will have to develop custom solution. From Application Insights standpoint, you may use Trace logs to record these details

2. Are there any services/configurations on Azure to log all requests/responses so that I don't need to modify the code?
No

3. What are the best practices for that kind of requirement? Is this ok to save all requests/responses into Application Insights to easily query/trace/investigate issues later or it should be somewhere else such as databases. etc. since the requests/responses may have sensitive info?
The choice of service will be up to you. You may use Application Insights trace logger to store it in AppInsights Workspace. Since this data is going to be unstructured, you may also use Azure storage OR Azure Cosmos DB. All the service mentioned have well defined .NET APIs that can help you achieve the goal

Please note, recording all the requests/responses from all the controllers will negatively impact the performance of your application. I would also like to understand the need for such a requirement. A stateful application with security as required should be able to provide you with most of the information required (For example - who performed what?)

Please let me know if you have any questions.

---

Please 'Accept as answer' and ‘Upvote’ if it helped so that it can help others in the community looking for help on similar topics.