Problem of StatusCode cannot be set.

Question

There are some troublesome crawlers that I want to make a middleware to counter them.

One of them always accesses some page that does not exist. I marked it by the status code and access times.

I got the status code by the code below:

public async Task Invoke(HttpContext context)
{
await _next.Invoke(context);
var statuscode=context.Response.StatusCode;
}

I don't why but if I place the var statuscode=context.Response.StatusCode; before the await _next.Invoke(context);, it always gets 200 status code no matter the page is exist.

If I place it after the await _next.Invoke(context);, it will get the right code. That's why I place it after the await _next.Invoke(context);.

Furthermore, the site will redirect to a reCAPTCHA validation page when the middleware detects the Ip maybe it is a crawler.

So, I modified the code as below:

public async Task Invoke(HttpContext context)
{
await _next.Invoke(context);
var statuscode=context.Response.StatusCode;
Boolean IsCrawler=false;
///some logic to detect the crawler
if(IsCrawler)
{
context.Response.StatusCode = 302;
context.Response.Headers["Cache-Control"] = "no-cache, no-store";
context.Response.Headers["Pragma"] = "no-cache";
context.Response.Headers["Expires"] = "-1";
context.Response.Redirect("");//URL of the reCAPTCHA validation page
return;
}
}

When the codes above are running locally, it won't report any exception.

However, after running on the sever computer, it always reports this error:
StatusCode cannot be set because the response has already started.

Why does the site response before the middleware is running completely? And how can I change the statuscode&header without the exception "StatusCode cannot be set because the response has already started."?

Thank you.

Answer 1

testing after other middleware leads to issues. the middleware (as in your case) may have already sent the responser headers back to the client. in your case you want to test for crawler and redirect or run more pipeline you code should be:

public async Task Invoke(HttpContext context)
{
    Boolean IsCrawler=false;
    ///some logic to detect the crawler
    if(IsCrawler)
    {
        return redirect instead of processing
        context.Response.StatusCode = 302;
        context.Response.Headers["Cache-Control"] = "no-cache, no-store";
        context.Response.Headers["Pragma"] = "no-cache";
        context.Response.Headers["Expires"] = "-1";
        context.Response.Redirect("");//URL of the reCAPTCHA validation page
        return;
    }
   // not crawler - run rest of middleware
   await _next.Invoke(context);
}