Have you tried deleting the public folder mailbox in EAC, and what result did you get?
What about referring to this: Cannot delete Public Folder that is flagged as Hold for Migration
Try using the following command to remove the PF mailbox:
Remove-Mailbox -PublicFolder -Identity "MyPF" -Force
Make sure the account you have the correct permission.