Azure User Impersonation does not work when User/Assignment enable on the Backend App Registration

Question

I have two application registrations: one for the backend and one for the frontend.

On the backend App Reg, I have defined (exposed) a scope called User_impersonation like the following:

On the frontend App Reg, I have added the permissions (User_impersonation) like the following:

Also on the frontend side, I enabled user/group assignment and assigned a security group; so that only the members of that security group can access/query this frontend.

So far, when a user queries the frontend with an access token (requested from the frontend app reg); everything works fine: the frontend app reg requests an access token from the backend and forwards the request of the user.

However, when I try to enable the user/group assignment on the backend app reg (by adding the same security group); the frontend app can no longer query the backend (User_impersonation fails); like the query times out with no reply from the backend.

Can someone help me make the User_impersonation work when the group assignment is enabled on both app regs (not only the frontend). Thanks.