@mattiast Thanks for posting in our Q&A.
For this issue, what you configured is correct. It seems something wrong with this feature. With Q&A limitation, it is suggested to create an online support to feedback this issue. Maybe there is a fix from backend. Here is the support link:
https://learn.microsoft.com/en-us/mem/get-support
Thanks for your understanding.
If the answer is helpful, please click "Accept Answer" and kindly upvote it. If you have extra questions about this answer, please click "Comment".
Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.