Is your security group converted from a distribution group? If yes, you could take a reference at the issue introduced here:
Exchange | Members can't remove themselves from security groups....
Please Note: Since the web site is not hosted by Microsoft, the link may change without notice. Microsoft does not guarantee the accuracy of this information.
However I tried the opeation in my environment convert my distribution group to a security group, and changed the attribute value wait for AD replication, it doesn't work.
So a workaround here, you could convert this security group to distribution group, then you will be able to perform the change name opertaion, after that, convert it back to SG again.
If an Answer is helpful, please click "Accept Answer" and upvote it.
Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.