Validity of Scan Reports Despite Logged Errors

Question

We ran the baseline scan and attack strategy scan against our model using the Red Team SDK. Although the output was generated, the logs show an error. Should I consider the report valid or invalid?

Answer 1

Hi Karthickumar Karuppiah,

Upon reviewing the provided report, it is my recommendation that it be considered invalid for the purposes of an actual red team assessment. Below are the key reasons supporting this conclusion:

• The logs indicate that HTTP 404 errors were encountered across all risk categories, meaning that the prompts could not be sent to the target system due to the unavailability of the Azure resource (project). As a result, no meaningful responses from the model were received or evaluated.
• The Attack Success Rate (ASR) for all tests is recorded as zero. However, this result should not be interpreted as a success of the model; rather, it reflects that the tests could not be executed due to backend errors, preventing any meaningful attack simulations.
• Although evaluation results are present, they are based on empty or failed conversations. These outputs do not reflect a legitimate red team assessment or testing of the target system.

Feel free to accept this as an answer.
Thankyou for reaching out to The Microsoft Q&A Portal