I think I have found the answer. The android client that works with ARM WVD is the beta one, where we need to opt in. The "Remote Desktop 8" client that the documentation links to actually uses the client ID of the old WVD app, hence does not work. I installed the beta and could then login from Android client fine.
It will be great to get the documentation updated to reflect this properly. I also find it curious that we need the beta version for a service that is GA, but it seems stable so less concerns (but I am sure customers will have concerns and will be quite right to).