For local AD domain-joined computers, you could use GPO to disable Cache Mode for Outlook Client. But those contractors' clients aren't domain-joined, we cannot manage it from GPO. The Cache Mode function cannot be managed from Exchange online server side.
By the way, I don't think it would make much sense even if we could disable cache mode for these clients. Because they could export data from mailbox when they could use their mailbox. In this way, they still could access emails after mailbox disabled/deleted.
For security reasons, you can disable these clients directly:
You can also let them use those clients, but delete all emails before deleting mailboxes, which will sync to all clients in use. But it still cannot prevent them back up their emails.
If the answer is helpful, please click "Accept Answer" and kindly upvote it. If you have extra questions about this answer, please click "Comment".
Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.