Set the permissions on the Default user principal on the Calendar folder to AvailabilityOnly:
Set-MailboxFolderPermission roomname:\calendar -User default -AccessRights AvailabilityOnly -WhatIf
This will not affect any delegates configured on the mailbox, as they get their own separate permission entry.