Hello @Dmytro ,
Most likely you are getting this failure due to your current AAD plan (supported/non supported features on it).
From what I can see with the screenshots you provided, it seems that you do not currently have an Azure AD Premium P1 or P2 license activated on your tenant. Group-based assignment and nested groups are currently only available with a P1 or P2 license.
Use a group to manage access to SaaS apps - Azure AD
and Azure Active Directory Pricing =>
Hope above answers your questions and concerns.
--------------------------------------------------------
Let us know if you need additional assistance. If the answer was helpful, please accept it and complete the quality survey so that others can find a solution.
Sincerely,
Olga Os