I found a solution for it:
I had to add the following file named "web.config" (which did not exist before) into the wwwroot of the App to make it work:
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<system.webServer>
<staticContent>
<remove fileExtension=".mp4" />
<mimeMap fileExtension=".mp4" mimeType="video/mp4" />
<clientCache cacheControlMode="UseMaxAge" cacheControlMaxAge="1.00:00:00" />
<remove fileExtension=".woff" />
<mimeMap fileExtension=".woff" mimeType="application/x-font-woff" />
<remove fileExtension=".json" />
<mimeMap fileExtension=".json" mimeType="application/json" />
</staticContent>
</system.webServer>
</configuration>
I did this with the App Service Editor, but it worked also when I added the file to my repository which gets build and deployed.
Btw, the "clientCache" xml tag sets the caching expiration to 1 day and is not required to make it work in the first place.
The app delivered the files as expected after I added the file.