You mean the scenario where the item was shared with a group? No, you cannot remove specific entries out of the expanded group membership, you can only remove the group entry itself. If you have sufficient permissions, try adjusting the actual group membership (https://learn.microsoft.com/en-us/graph/api/group-delete-members?view=graph-rest-beta&tabs=http).
Also note that you cannot remove inherited entries, ones with non-null value of inheritedFrom.