Use the RBAC approach - rather than using access control policies to assign permissions to the vault artifacts. With RBAC, you can assign permissions to individual secrets. Access policies are vault-wide - so if you want to set permissions to individual secrets, you'd need to place them in different vaults.
More at https://learn.microsoft.com/en-us/azure/key-vault/general/rbac-access-policy
For implementation details, refer to https://learn.microsoft.com/en-us/azure/key-vault/general/rbac-guide?tabs=azure-cli
If the above response helps answer your question, remember to "Accept Answer" so that others in the community facing similar issues can easily find the solution. Your contribution is highly appreciated.
hth
Marcin