You should switch to using the Graph SDK for PowerShell, as the Azure AD module is not receiving any updates. You should also use server-side filters where appropriate, in this case you can do something like this:
Select-MgProfile beta
$users = Get-MgUser -Filter "userType eq 'Guest' and externalUserState eq 'PendingAcceptance'" -All -ExpandProperty memberOf
$users | ? {!($_.MemberOf.Id -match 'c91cd116-a8a5-443b-9ae1-e1f0bade4a23|528a4052-fa6c-4495-b39f-2820f8e1e8db')}
where you have to provide the GUIDs of the groups you want to exclude.