I figured out the answer - it ended up being an example on quotas in MS documentation
By putting an increment condition on the quota, I can limit requests being counted to those with particular response status codes
This one limits any particular IP address on any one API to 30 invalid request calls in a 5 minute (300 second) period. Unfortunately, APIM won't allow smaller interval than 5 minutes for the renewal-period
<quota-by-key calls="30" renewal-period="300" counter-key="@(context.Request.IpAddress + context.Operation.Id)" increment-condition="@(context.Response.StatusCode > 202 && context.Response.StatusCode != 401 && context.Response.StatusCode != 403)" />