@Kai Vermeegen
Thank you for your question! I wasn't able to replicate your issue, since both my AzureAD and Guest user were able to see my Key Vault's secrets. However, are you able to provide the RBAC roles that you're assigning to the group and at what scope those RBAC roles apply to? i.e. resource group, key vault, etc. Any screenshots would be helpful as well so I can gain a better understanding of your issue.
Findings:
Trying to replicate your issue, I created a group called "KeyVaultAccess" and added a member and guest user to this group.
I assigned reader rights to my Key Vault's resource group for the entire group.
Finally I added an access policy to only allow for the group to have Secret Management rights.
Note: Both users can see all my key vaults in the resource group because of the Reader RBAC assignment. However, since access to a key vault is controlled through two interfaces: the management plane (RBAC) and the data plane (access policies), both of my users can only execute secret operations because of the Secret Management permissions I assigned.
I also found that you can use Azure Cloud Shell/CLI so that the Guest user can list secrets (without the Reader assignment).
Connect-AzAccount -Tenant "AzureAD Tenant ID"
az keyvault secret list --vault-name "KVname"
I hope this helps, if you have any other questions, please let me know.
Thank you for your time and patience throughout this issue.