Well, you can use this command and see how it works for you. If it does, awesome! If not, then go the Outlook route. The reason I mention it is because its the only sure way I have that works consistently!
Clearing the SharingPermissionFlags removes the delegate feature.
Example:
Set-MailboxFolderPermission huku:\calendar -User vasil -AccessRights Editor -SharingPermissionFlags None
https://www.michev.info/Blog/Post/1976/managing-outlook-delegates-via-powershell