Gatsby in Azure Static Web app and Client only routes loads the wrong css and does not display the 404 page

Question

I have a Gatsby site hosted with Azure's static web app with the following configuration:

staticwebapp.config.json:

{
  "navigationFallback": {
    "rewrite": "/index.html"
  },
  "responseOverrides": {
        "404": {
          "rewrite": "/404.html"
        }
    }
}

But if you link directly to the login page (which is a client-only route created with @reach-router) or refresh it while you are on it doest if redirect you briefly to / then to /user/login. However the styling is a mess, pictures arent being displayed, and if you try to go to a route which doesn't exist will you again be redirected to the index page and not the "custom" 404 page.
It seems like the "navigationFallback" is overriding the "responseOverrides".