Avoid error log for payloads over 102,400 bytes request/response payload

Question

Avoid error log for payloads over 102,400 bytes request/response payload

akhilas@secful.com 20

We enable the error logging in APIM.But we need to avoid error log for payloads over 102,400 bytes request/response payload.is it possible for dynamically restrict logging?

User's image

akhilas@secful.com 20 Reputation points

2024-01-03T04:13:39.8766667+00:00
1.)I am using the validate-content policy

<validate-content unspecified-content-type-action="ignore" max-size="102400" size-exceeded-action="detect" errors-variable-name="Error" />

If an error is encountered with the validate-content policy, we aim to avoid full log writing.

To check if the request body size exceeds 100KB before processing and avoid loading it into memory.Could you give which is better validate-content policy or we can use the <choose> policy with a condition based on the Content-Length header?

<choose> <when condition="@(context.Request.Body.Length <= 1048576)">

Accepted answer

0 additional answers

Your answer

akhilas@secful.com 20 Reputation points

2024-01-03T04:13:39.8766667+00:00

1.)I am using the validate-content policy

<validate-content unspecified-content-type-action="ignore" max-size="102400" size-exceeded-action="detect" errors-variable-name="Error" />

If an error is encountered with the validate-content policy, we aim to avoid full log writing.

To check if the request body size exceeds 100KB before processing and avoid loading it into memory.Could you give which is better validate-content policy or we can use the <choose> policy with a condition based on the Content-Length header?

<choose> <when condition="@(context.Request.Body.Length <= 1048576)">

Answer 1

MuthuKumaranMurugaachari-MSFT 22,441 Moderator

******@secful.com Thanks for posting your question in Microsoft Q&A. You can specify Number of payload bytes to log to a value (up to 8192) other than 0 and this can be set individually for front end, and backend request/response. Check out doc: Diagnostics logs settings reference: API Management for more info.

User's image

However, the limit is for the initial bytes to log and not possible to avoid error log entirely. If you wish this feature, please submit feedback via https://aka.ms/apimwish and others with similar interests can upvote your idea. This will help our product team prioritize the features.

I hope this helps and let me know if any questions.

If you found the answer to your question helpful, please take a moment to mark it as Yes for others to benefit from your experience. Or simply add a comment tagging me and would be happy to answer your questions.

akhilas@secful.com 20 Reputation points

2024-01-03T04:17:54.43+00:00
Hi @MuthuKumaranMurugaachari-MSFT

1.)I am using the validate-content policy

<validate-content unspecified-content-type-action="ignore" max-size="102400" size-exceeded-action="detect" errors-variable-name="Error" />

If an error is encountered with the validate-content policy, we aim to avoid full log writing.

To check if the request body size exceeds 100KB before processing and avoid loading it into memory.Could you give which is better validate-content policy or we can use the <choose> policy with a condition based on the Content-Length header?

<choose> <when condition="@(context.Request.Body.Length <= 1048576)">
MuthuKumaranMurugaachari-MSFT 22,441 Reputation points Moderator

2024-01-03T14:39:53.7933333+00:00

******@secful.com Can you elaborate how you have configured logging, where it is sent to (Azure Monitor or App Insights), Number of payload bytes to log value? The screenshot shows you have set it in All API scope.

validate-content policy is commonly used to validate against schemas, and it would be simpler to use length of the body with the policy expression like you described. However, with Always log errors setting, all failures will be logged and wondering what action you are looking to perform via policy.

If you can share additional details requested above, that will be helpful to understand the set up.
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
akhilas@secful.com 20 Reputation points

2024-01-03T19:49:18.5666667+00:00

@MuthuKumaranMurugaachari-MSFT

i am using App Insights.Do not generate an error log for payloads over 102,400 bytes, as this is causing their log space to be filled up.We use a validate-content policy that restricts content to a maximum of 100KB; exceeding this limit results in a "PayloadTooLarge" error. Although this error is known, we currently do not display logs or provide options to proactively prevent the occurrence of the "Payload too large" error. The system can handle payloads larger than 100KB through custom logic, and it successfully returns results with a status of 200. However, the issue lies in the logs, where an exception related to this large payload is being recorded. Could you suggest any option?
akhilas@secful.com 20 Reputation points

2024-01-03T20:06:13.21+00:00

@MuthuKumaranMurugaachari-MSFT

i am using App Insights.Do not generate an error log for payloads over 102,400 bytes, as this is causing their log space to be filled up.We use a validate-content policy that restricts content to a maximum of 100KB; exceeding this limit results in a "PayloadTooLarge" error. Although this error is known, we currently do not display logs or provide options to proactively prevent the occurrence of the "Payload too large" error. The system can handle payloads larger than 100KB through custom logic, and it successfully returns results with a status of 200. However, the issue lies in the logs, where an exception related to this large payload is being recorded. Could you suggest any option?
MuthuKumaranMurugaachari-MSFT 22,441 Reputation points Moderator

2024-01-03T21:08:03.19+00:00

******@secful.com I think there is some confusion and let me clarify.

When you enable diagnostic settings to send it to App Insights (All APIs scope) with Always log errors setting, all error logs will be sent to App Insights and max payload value is up to 8192 as described in Application Insights doc. The payload with greater size will not be fully logged in App Insights (only initial bytes up to the max value).

If you are looking to stop this error logging in App Insights, it is not possible based on a condition.

However, with validate-content policy, you can restrict max-size (Maximum allowed value: 102,400 bytes) and based on size-exceeded-action, action will be determined.

With action detect, it is just going to log validation errors to variable defined in errors-variable-name and you may log this error info to App Insights using trace policy. But this is also propagated to context.LastError property and will be logged in App Insights like you described with PayloadTooLarge error. This is as per design and cannot be modified.

You may choose to stop processing the request and return as error with Prevent action so that the client app is aware about the payload limitation.

Let me know if any questions.
akhilas@secful.com 20 Reputation points

2024-01-04T17:55:52.7633333+00:00

Thank you for your valuable assistance. Could you suggest any option to use insted of validate-content validates the size of the payload body.
MuthuKumaranMurugaachari-MSFT 22,441 Reputation points Moderator

2024-01-04T18:20:04.43+00:00

******@secful.com I have sent an email (shared in feedback) and we can connect offline to understand more about your scenario.
MuthuKumaranMurugaachari-MSFT 22,441 Reputation points Moderator

2024-01-05T18:11:38.1766667+00:00

******@secful.com If you have any issues receiving email that I sent, you can also reach out to us via azcommunity@microsoft.com referencing this thread with your subscription ID (not name), APIM details and subject as "Attn: Muthu". I can assist you offline.

Share via

Avoid error log for payloads over 102,400 bytes request/response payload

0 additional answers

Your answer