Hi Oscar,
Thank you for posting your query on Microsoft Q&A!
I can reproduce the same behaviour in my own test tenant. I will report this internally to request a fix but it may take some time. This toggle is not designed to be a security measure but more to prevent users from mis-configuring their profile or apps etc..
I would recommend creating a Conditional Access policy which blocks non-admin users from accessing the Windows Azure Service Management API, which would also prevent access with PowerShell etc..
We have some more details on this here - https://learn.microsoft.com/en-us/entra/fundamentals/users-default-permissions#restrict-member-users-default-permissions
Let me know if you have any further queries, I would be happy to help.
Please "Accept the answer" if the information helped you. This will help us and others in the community as well.
Kind Regards,
Donal