Sorry for the late reply, I checked the azure portal again right now.
In the app configuration, the secrets now show up correctly (after such a long time, I didn't change the configuration since then):
Then, I tested the app - still has access(!).
After that, I restarted the app -> no access anymore.
The questions is then:
Why is access not revoked immediately after removing the access policy? It seems to take hours or even days to come into effect and only an app restart then actually denies access. This seems a potentially significant security risk.